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Preface 

Increasingly  powerful  pharmaceuticals  are  of  increasing  cHnical  and  economic  importance  to 
managed  care  organizations.  Expenditures  on  pharmaceuticals  have  also  been  increasing,  a  trend 
driven  by  a  number  of  factors,  including  the  accelerating  development  of  more-innovative  and  more- 
expensive  agents;  rising  pharmaceutical  prices;  and  higher  utihzation  due  to  the  aging  of  the 
population,  direct-to-consumer  advertising,  and  other  factors.  More  often  today,  health  plans  and 
provider  organizations  are  responsible  for  managing  and  paying  for  these  increasing  pharmaceutical 
costs,  which  creates  an  incentive  for  them  to  go  beyond  a  focus  on  chnical  effectiveness  and  safety  to 
evaluate  the  cost-effectiveness  of  new  drugs.  In  addition,  the  development  of  drug  formularies  and 
the  current  focus  on  best  practices  require  that  each  new  drug  be  assessed  relative  to  available 
alternatives.  The  formal  controls  and  guideHnes  resulting  from  managed  care  processes  can 
increase  quality  and  cost  efficiency,  but  can  also  be  a  barrier  to  desirable  innovations.  Together, 
these  developments  have  caused  managed  care  organizations  (MCOs)  to  reahze  that  making  good 
decisions  on  new  pharmaceuticals  is  to  their  immediate  financial  and  chnical  benefit.  Accordingly, 
many  have  expressed  interest  in  improving  their  abihty  to  evaluate  new  pharmaceuticals. 

Four  interrelated  developments  in  today's  health  care  system  provided  the  motivation  for 
developing  the  framework  for  pharmaceutical  technology  assessment  reported  here: 


•  The  rising  importance  of  pharmaceuticals  in  health  care,  and  the  prospect  for  increasing 
numbers  of  more  costly  new  entities,  including  some  that  will  affect  the  ciirrent  pattern  of  care 
by  creating  demand  for  treatment  or  substituting  for  other  services  such  as  hospital  care  or 
surgery. 

•  The  increasingly  wide  range  of  managed  care  organizations,  including  health  plans  and  provider 
organizations,  held  responsible  by  purchasers  for  managing  and  paying  for  pharmaceuticals. 

•  The  potential  for  the  quahty-improvement,  guidehne,  and  disease-management 
movements — ^which  codify  current  chnical  practice — to  affect  the  diffusion  of  technology. 

•  The  widely  perceived  (by  government,  third-party  payers,  MCOs,  pharmaceutical  companies, 
etc.)  need  for  better  tools,  processes,  and  support  systems  to  help  managed  care  organizations 
evaluate  new  pharmaceutical  technology  for  possible  health  plan  coverage  or  patient  use. 


This  report  describes  the  processes  by  which  managed  care  organizations  evaluate 
pharmaceutical  technologies  and  suggests  pathways  for  organizing  improvement.  It  reviews  current 
evaluation  procedures,  provides  a  framework  for  future  evaluations,  introduces  evaluation 
approaches,  and  annotates  resources  for  further  research  on  each  topic  covered.  The  report  is 
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intended  to  assist  managed  care  organizations,  including  health  plans,  medical  groups,  pharmacy 
benefit  management  firms,  and  others  interested  in  initiating  formal  pharmaceutical  evaluation 
activities  or  in  improving  existing  procedures. 

The  report  is  based  on  research  conducted  by  RAND  Health.  RAND  Health  furthers 
RAND’s  mission  of  helping  improve  policy  and  decisionmaking  through  research  and  analysis,  by 
working  to  improve  health  care  systems  and  advance  understanding  of  how  the  organization  and 
financing  of  care  affect  costs,  quality,  and  access. 
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Executive  Summary 

Increasingly  powerful  pharmaceuticals  are  of  increasing  cHnical  and  economic  importance  to 
managed  care  organizations.  Expenditures  on  pharmaceuticals  have  also  been  increasing,  a  trend 
driven  by  a  number  of  factors,  including  the  accelerating  development  of  more-innovative  and  more- 
expensive  agents;  rising  pharmaceutical  prices;  and  higher  utilization  due  to  the  aging  of  the 
population,  direct-to-consumer  advertising,  and  other  factors.  More  often  today,  health  plans  and 
provider  organizations  are  responsible  for  managing  and  paying  for  these  cost  increases,  which 
creates  an  incentive  for  them  to  go  beyond  a  focus  on  chnical  effectiveness  and  safety  to  evaluate  the 
cost-effectiveness  of  new  drugs.  In  addition,  the  development  of  drug  formularies  and  the  current 
focus  on  best  practices  require  that  each  new  drug  be  assessed  relative  to  available  alternatives. 
The  formal  controls  and  guidelines  resulting  from  managed  care  processes  can  increase  quahty  and 
cost  efficiency,  but  can  also  be  a  barrier  to  desirable  innovations.  Together,  these  developments 
have  caused  managed  care  organizations  to  reahze  that  making  good  decisions  on  new 
pharmaceuticals  is  to  their  immediate  financial  and  clinical  benefit.  Accordingly,  many  have 
expressed  interest  in  improving  their  ability  to  evaluate  new  pharmaceuticals. 

Four  interrelated  developments  in  today’s  health  care  system  provided  the  motivation  for 
developing  the  framework  for  pharmaceutical  technology  assessment  reported  here: 


•  The  rising  importance  of  pharmaceuticals  in  health  care,  and  the  prospect  for  increasing 
numbers  of  more-costly  new  entities,  including  some  that  will  affect  the  current  pattern  of  care 
by  creating  demand  for  treatment  or  by  substituting  for  other  services  such  as  hospital  care  or 
surgery. 

•  The  increasingly  wide  range  of  managed  care  organizations,  including  health  plans  and  provider 
organizations,  held  responsible  by  purchasers  for  managing  and  paying  for  pharmaceuticals. 

•  The  potential  for  the  quahty-improvement,  guideline,  and  disease-management 
movements — which  codify  current  cUnical  practice — ^to  affect  the  diffusion  of  technology. 

•  The  widely  perceived  (by  government,  third-party  payers,  MCOs,  pharmaceutical  companies, 
etc.)  need  for  better  tools,  processes,  and  support  systems  to  help  managed  care  organizations 
evaluate  new  pharmaceutical  technology  for  possible  health  plan  coverage  or  patient  use. 


This  report  describes  the  processes  by  which  managed  care  organizations  evaluate 

pharmaceutical  technologies  and  suggests  pathways  for  organizing  improvement.  It  reviews  current 

procedures,  provides  a  framework  for  future  evaluations,  introduces  approaches  to  evaluation  within 

that  framework,  and  annotates  resources  for  further  research  on  each  topic  covered.  The  report  is 
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intended  to  assist  managed  care  organizations,  including  health  plans,  medical  groups,  pharmacy 
benefit  management  firms  (PBMs),  and  others  interested  in  initiating  formal  pharmaceutical 
evaluation  activities  or  in  improving  existing  procedures. 

The  data  collection  component  of  our  study  consisted  of  a  two-part  process  of  interviewing 
representatives  of  relevant  organizations  and  reviewing  the  relevant  hterature.  We  performed  semi- 
structured  quahtative  interviews  of  decisionmakers  at  12  national  and  regional  managed  care 
organizations,  at  medical  groups  at  some  risk  for  their  pharmacy  costs,  and  at  pharmacy  benefit 
management  firms.  Our  hterature  review  uncovered  over  1,000  articles,  which  we  narrowed  to  38 
for  a  thorough  review.  We  then  synthesized  the  results  of  the  interviews  and  literature  review  to 
produce  a  review  of  current  practices,  using  five  questions  to  frame  the  synthesis: 


•  Who  decides  to  accept  or  reject  new  pharmaceutical  technology,  and  how? 

•  What  decision  options  (e,g.,  **Exclude  drug,'*  ^'Assign  to  high-cost  tier," and  *Ttequire prior 
authorization ')  are  considered  ? 

•  What  factors  affect  the  timing  and  resources  devoted  to  pharmaceutical  technology  assessment? 

•  What  types  of  evaluations  are  performed  and  what  types  of  evidence  do  they  use? 

•  How  are  decisions  implemented  and  the  results  monitored? 


This  review  informed  the  second  part  of  the  study,  in  which  we  develop  a  prescriptive 
framework  for  pharmaceutical  technology  assessment  and  a  guide  to  the  major  relevant  analytic 
approaches.  The  appendices  to  this  report  provide  an  annotated  bibliography  and  a  copy  of  the 
interview  questionnaire. 

Our  review  of  current  practices  found  wide  variation  in  the  process  whereby  organizations 
assess  pharmaceutical  technology  and  manage  pharmacy  benefits.  The  locus  of  pharmaceutical 
evaluation  varies  by  organizational  type,  as  does  range  of  decision  options  available  within  each 
organization.  The  timing  of  the  assessment  appears  to  be  a  function  of  the  organization’s  reason  for 
performing  the  assessment.  The  process  used  by  the  decisionmakers  we  interviewed  is  often  neither 
reproducible  nor  rigorous,  but  the  level  of  rigor  does  appear  to  increase  with  the  level  of  controversy 
or  demand  for  the  drug.  Finally,  the  emphasis  of  the  evaluations  is  usually  on  safety,  efficacy,  and 
direct  drug  costs  rather  than  on  overall  potential  financial  impact  or  health  outcomes,  although  all 
respondents  were  concerned  with  impact  and  outcomes. 

Those  managed  care  organizations  with  whom  we  spoke  wanted  a  structured  process,  or 
framework,  for  assessing  pharmaceutical  technology,  believing  that  use  of  such  a  framework  would, 
over  time,  increase  the  ease  of  sharing  information,  the  rationality  of  the  decisionmaking  process, 
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and  the  transferabihty  of  best  practices.  However,  some  of  our  interviewees  cautioned  that  for  a 
structured  guideline  to  be  helpful  in  the  “real  world  of  managed  care,”  it  would  need  to  be  flexible, 
efficient,  and  adaptable  by  diverse  organizational  structures  and  for  a  variety  of  purposes. 

With  that  comment  in  mind,  and  in  response  to  the  other  findings  in  this  report,  we  suggest 
a  structured  process  for  assessment  of  pharmaceutical  technology  in  managed  care  that  includes  the 
following  steps: 


•  Characterize  the  drug  and  the  disease  by  identifying  potential  cfinical  risks  and  benefits  of  the 
drug,  describing  the  drug  in  relation  to  the  marketplace,  and  assessing  the  importance  of  the 
disease  in  terms  of  its  current  and  potential  future  prevalence  and  impact. 

•  Characterize  the  decisionmaking  organization  in  terms  of  the  extent  to  which  and  means  by 
which  it  can  promote  or  Umit  the  use  of  the  pharmaceutical  in  question. 

•  Define  the  decision  options  by  identifying  (but  not  yet  choosing  among)  the  available 
organizational  processes  that  can  be  used  to  control  the  prescribing  and  use  of  the  drug. 

•  Define  the  method  needed  to  make  the  decision,  including  rigor  and  resources  required,  the  agent 
who  will  perform  the  assessment,  the  timing  of  the  assessment,  and  the  most  appropriate 
methodologies. 

•  Perform  the  assessment  according  to  standard  approaches  to  systematically  obtaining  expert 
opinion,  performing  evidence  assessment,  or  constructing  a  mathematical  model,  whichever  is 
selected. 

•  Make,  implement,  and  evaluate  the  decision. 


In  summary,  most  managed  care  organizations  have  internal  procedures  for  assessing  new 
pharmaceutical  technologies.  Medical  directors  in  a  variety  of  organizations  commonly  beheve  these 
procedures  to  be  less  rigorous  and  less  systematic  than  are  desirable  for  making  decisions  having 
substantial  implications  for  the  short-  or  long-term  health  of  the  organization  and  its  patients.  The 
decisionmaking  fi*amework  outhned  above  and  the  techniques  incorporated  in  it  describe  an 
objective,  reproducible  process  for  approaching  these  analyses.  Not  a  cookbook,  it  is  instead  a  “road 
map”  for  an  approach  that  can  help  an  organization  expose  hidden  biases  in  cmrent  processes, 
build  on  its  experiences  with  evaluation,  and  reduce  the  risk  of  illogical  or  poorly  thought-through 
decisions. 
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1.  Introduction 


Purpose  of  This  Report 

The  introduction  of  new  drugs,  or  pharmaceutical  technologies,  over  the  past  decade  has 
required  managed  care  organizations  to  perform  a  balancing  act.  On  the  one  hand,  this  rapid 
progress  in  drug  therapy  has  improved  well-being  and  decreased  the  need  for  intrusive  and 
expensive  services  in  previously  unimaginable  ways.  On  the  other  hand,  the  cost  of  new  drugs  has 
strained  pharmacy  budgets  and,  even  though  the  effect  on  overall  costs  varies,  has  often  forced 
providers  and  p\xrchasers  to  balance  the  competing  demands  of  cost,  effectiveness,  and  access. 

Under  managed  care  arrangements,  health  plans  and  provider  organizations  bear 
substantial  cUnical  and  financial  responsibifity  for  pharmaceutical  use.  However,  these 
organizations  do  not  control  pharmaceutical  use,  which  is  actually  driven  by  the  behavior  of 
enroUees  and  the  prescribing  patterns  of  their  physicians.  As  a  result,  organizations  have  developed 
a  variety  of  strategies  for  influencing  pharmaceutical  use:  fixed  or  variable  co-payments,  prior 
authorization,  encouragement  of  drug  substitution,  mandatory  step  therapies,^  dosing  or  duration 
hmits,  and  exclusion  of  particular  classes  of  drugs  from  coverage  (Schweitzer,  1997). 

Although  these  strategies  are  usually  aimed  at  controlling  expenditures,  organizations  also 
employ  them  to  encourage  the  use  of  drugs  that  provide  clinical,  economic,  or  organizational 
advantages.  As  advances  in  molecular  medicine  and  human  genetics  continue  to  create  increasingly 
powerful  but  difficult-to-use  “designer  drugs,”  strategies  for  encouraging  use  may  be  required  more 
often  to  optimize  chnical  outcomes  and  manage  costs.  However,  new  drugs,  such  as  Norplant  (left 
under  the  skin  for  prolonged  periods),  artificial  joint  fluid  (which  is  injected  directly  into  the  knee),  or 
vectors  for  dehvering  therapeutic  genes,  may  require  novel  delivery  systems  and  may  have  costs  that 
are  less  like  those  of  conventional  pharmaceuticals  and  more  like  those  of  traditional  hospital  care. 
Thus,  it  is  often  important  to  fully  understand  the  cost  and  quality  implications  of  new 
pharmaceuticals  before  accepting  or  rejecting  them. 

Managed  care  organizations  in  particular  need  to  understand  how  various  pharmaceutical 
pohcies  and  procedures  will  govern  the  effect  of  a  new  drug.  Will  requiring  pre-authorization 
preferentially  reduce  inappropriate  use  of  a  particular  drug?  Will  step-therapy  programs  ensure 
that  an  effective  but  potentially  toxic  drug  is  used  only  after  first-line  drugs  are  exhausted?  Will 
assigning  a  co-payment  reduce  adherence  to  a  drug  regimen  among  patients  who  may  benefit  most 
fi-om  it? 


1  In  step  therapy,  certain  therapeutic  substitutes  are  tried  before  a  more  potent,  toxic,  and/or  expensive 
drug  is  prescribed. 
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Why  Pharmaceutical  Technology  Assessment  Is  Important 

Answering  these  questions  and  making  good  decisions  on  the  use  of  new  pharmaceuticals  is 
financially  and  clinically  responsible,  and  is  therefore  in  the  short-  and  long-term  interest  of 
managed  care  organizations.  These  decisions  are  significant  because  increasingly  powerful 
pharmaceuticals  are  of  increasing  chnical  and  financial  importance  to  health  care  and  health  care 
organizations.  Expenditures  on  pharmaceuticals  are  rising,  to  the  increasing  concern  of  managed 
care  organizations,  health  care  providers,  purchasers,  and  consumers.  In  1998,  the  retail 
pharmaceutical  market  in  the  United  States  generated  $93.7  billion  on  a  total  volume  of  2.4  bilHon 
prescriptions,  representing  a  4%  increase  in  the  number  of  prescriptions  in  one  year  (Novartis, 

1999).  Meanwhile,  prescription  drug  expenditures  in  the  fourth  quarter  of  1998  increased  by  16.8% 
over  the  same  period  in  1997  (Winslow,  1999).  At  many  health  plans,  prescription  drugs  now 
account  for  11  to  14%  of  total  medical  expenses,  up  from  7%  in  the  early  1990s  (Winslow,  1999),  For 
diseases  requiring  “high-tech”  treatment,  the  increases  have  been  more  dramatic.  For  example, 
drug  costs  per  patient  per  month  rose  38%  in  the  two  years  after  the  introduction  of  new  highly 
active  combination  treatments  for  HIV  (Joyce,  1999).^  Researchers  for  the  Health  Care  Financing 
Administration  (HCFA)  are  predicting  that  these  trends  will  continue,  with  prescription  drug  costs 
increasing  nearly  10%  every  year  through  2007  (Hagland,  1998).  Health  maintenance  organizations 
(HMOs)  are  predicting  a  20%  increase  in  annual  drug  costs,  even  though  overall  health  care  costs 
are  expected  to  rise  only  5%  in  2000  (Galewitz,  1999). 

Increased  expenditures  are  driven  by  both  increases  in  the  quantity  of  drugs  prescribed  and 
increases  in  the  unit  prices  of  drugs.  The  National  Association  of  Chain  Drug  Stores  predicts  that 
the  number  of  prescriptions  filled  in  pharmacies  will  rise  and  that  the  dollars  spent  on  prescriptions 
will  rise  even  more  (Harris,  1999).  The  average  cost  per  prescription  rose  7%,  to  $38,  in  1998,  and 
certain  new  biotechnology  drugs  can  run  as  high  as  $14,000  per  year  for  individuals  with  common 
chronic  conditions  (Novartis,  1999;  Winslow,  1999). ^  There  are  two  readily  identifiable  causes  of 
increasing  unit  prices  in  the  U.S.  market,  but  the  relative  importance  of  these  two  factors  has  been 
the  subject  of  considerable  debate  (Schweitzer,  1997).  First  is  the  possibility  that  increasing  R&D 
expenses  and  shareholder  demands  are  pressuring  drug  manufacturers  to  price  products  in  the 
largely  unregulated  U.S.  market  to  compensate  for  lower  margins  elsewhere.  Second  is  that  prices 
have  risen  in  response  to  increasing  patient  demand  for  old  and,  especially,  newer  drugs,  which  are 
generally  more  costly  than  the  drugs  that  they  replace  or  which  target  new  diseases. 


2  The  total  cost  of  HIV  care  per  patient  per  month  dechned  over  this  period  as  a  result  of  declines  in 
hospital  care,  a  fact  that  underscores  the  need  for  comprehensive  assessments. 

3  This  is  by  no  means  the  hmit  on  cost.  For  example,  the  annual  cost  of  the  orphan  drug  alglucerase 
(Ceredase)  for  the  rare  inherited  condition  Gaucher’s  disease  is  many  times  as  expensive. 
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Innovative  drugs — ^including  pharmaceuticals  to  treat  conditions  that  were  previously 
untreatable,  treatable  only  through  hospitalization  or  smgery,  or  perhaps  even  not  considered 
appropriate  for  medical  therapy — ^tend  to  he  more  costly.  Significant  new  treatments  are  now 
available  for  conditions  such  as  AIDS,  hepatitis  C,  breast  cancer,  and  transplant  rejection.  Some  of 
them  produce  offsetting  savings  from  avoided  morbidity — for  example,  cholesterol-lowering  agents 
that  reduce  heart  attack  rates  or  by  substituting  for  more-expensive  treatment  modahties — ^for 
example,  pharmacotherapy  replaces  chronic  hospitahzation  for  psychosis  and  surgery  for 
gastrointestinal  ulcers  in  most  patients.  Some  other  types  of  new  drugs  are  unlikely  to  produce 
offsetting  savings  over  the  short  term.  These  include  new  therapies  for  conditions  that  were 
previously  untreatable  or  for  which  drug  therapy  was  poorly  tolerated — ^for  example,  Cox  II 
inhibitors  for  pain.  Finally,  the  costs  of  “lifestyle”  drugs  for  conditions  that  are  not  imiversally 
regarded  as  diseases — ^for  example,  male  pattern  baldness  or  moderate  obesity — are  hkely  to 
translate  directly  into  increased  overall  costs.  Therefore,  the  effect  of  new  drug  use  on  overall 
expenditures  is  variable. 

Other  factors  are  increasing  the  demand  for  pharmaceuticals.  For  example,  the  “baby 
boom”  generation  is  aging,  causing  the  prevalence  of  chronic  diseases  such  as  diabetes  to  rise 
(Hagland,  1998).  The  number  of  persons  in  treatment  for  some  diseases  has  been  increased  by 
disease-management  programs.  As  part  of  the  drive  to  improve  outcomes  while  controlling  overall 
costs  (Epstein  and  Sherwood,  1996),  such  programs  may  systematically  identify  persons  at  risk  in  a 
population  under  care — ^for  example,  people  likely  to  get  diabetes — ^and  intervene  with  a 
coordinated,  comprehensive  program  of  care  along  the  continuum  of  the  disease — ^for  example, 
monitoring  diet  and  laboratory  work  for  mUd  diabetes  and  adding  pharmaceuticals  if  the  disease 
worsens.  In  this  way,  many  disease-management  programs  encourage  the  use  of  pharmaceutical 
therapies  at  an  earlier  stage  of  disease  than  is  usual  (Epstein  and  Sherwood,  1996;  EUrodt  et  al., 
1997). 

The  preferences  of  increasingly  informed  consumers  are  also  dramatically  affecting  demand. 
One  major  reason  for  this  is  the  rise  of  direct-to-consumer  advertising,  which  has  increased  patient 
awareness  and  demand  for  specific  drugs.  Spending  on  such  advertising  increased  firom  essentially 
nil  10  years  ago  to  $104  million  in  1993,  $600  million  in  1996,  and  $600  miUion  in  just  the  first  half 
of  1998  (Nordenberg,  1998;  Hagland,  1998).  The  growth  of  the  Internet  has  also  made  health 
information  more  accessible  to  consumers.  A  1999  telephone  survey  estimated  that,  of  the  88 
miUion  Americans  with  access  to  the  Internet,  68%  have  used  the  World  Wide  Web  to  access  health 
care  information  (Louis  Harris  and  Associates,  1999). 

These  increases  in  expenditures  directly  argue  for  better  decisions  on  pharmaceutical 
technologies  because  payers  are  increasingly  holding  health  plans  and  provider  organizations 
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accountable  for  the  costs  of  care.  More  and  more  provider  organizations  are  entering  capitation  and 
other  risk’sharing  arrangements,  so  they  can  no  longer  have  others  billed  for  the  costs  of  their 
clinical  decisions.  Consequently,  prescribers  are  under  mounting  pressure  to  move  beyond 
traditional  safety  and  efficacy  concerns  to  consider  cost-effectiveness  and  other  pharmacoeconomic 
factors,  such  as  ease  of  administration.  Moreover,  as  purchasers  of  health  care  become  more 
sophisticated,  managed  care  organizations  are  increasingly  required  to  measure  and  monitor  the 
outcomes  of  care  and  to  use  this  information  to  improve  care  processes.  To  the  extent  that  the 
relationship  between  medical  interventions  and  disability  becomes  better  understood  in  the  future, 
organizations  may  also  be  held  accountable  for  the  productivity  and  labor  force  participation  of  their 
patients.  To  meet  these  new  demands,  provider  organizations  have  asked  for  practical  models  for 
assessing  the  potential  effects  of  new  drugs,  both  financially  and  clinically.  These  models  should 
account  for  not  only  the  outcome  of  a  course  of  treatment  with  a  particular  drug  but  also  its  effect  on 
the  entire  care  process  and  beyond.  Section  3  describes  several  approaches  to  generating  such 
models. 

Health  plans  and  providers  are  being  held  more  responsible  for  measurable  outcomes  of 
care.  Quahty  management,  whose  tools  include  clinical  guidelines  and  protocols  (“care  paths,”  for 
example)  as  means  of  codifying  “best  practices,”  require  good  information  on  new  drugs.  They  are 
intended  to  act  hke  disease-management  programs  in  promoting  best  practices  in  drug  therapy. 
However,  guidelines  can  be  static,  thereby  creating  de  facto  barriers  to  improvements  in  the  process 
of  care.  Severe  cost  pressures  and  thinning  margins  can  exacerbate  this  barrier  effect  by  reducing 
the  appeal  of  new  technologies  that  create  initial  cost  increases.  The  danger  is  that  if  new 
technologies  that  improve  health  and  quafity  of  life  are  not  accepted  and  implemented,  limited 
receptivity  to  innovation  in  the  short  run  can  create  greater  costs  in  the  end.  Systematic  approaches 
to  evaluation  that  make  it  easier  to  assess  the  complete  impact  of  the  drug  and  predict  the  longer- 
term  effects  of  using  a  new  agent  can  help  to  overcome  these  barriers  to  change.  Even  though  longer- 
term  effects  are  currently  seen  as  more  important  to  society  generally  than  to  individual  plans,  we 
would  argue  that  all  plans  would  benefit  if  longer-term  thinking  becomes  more  common.  Moreover, 
longer-term  thinking  may  provide  individual  plans  with  a  strategic  advantage  in  the  marketplace. 

Pharmaceuticals  As  a  Technology  to  Be  Assessed 

To  guide  the  conduct  of  the  study,  we  looked  to  the  perspective  of  technology  assessment. 
One  accepted  definition  of  technology  is  “the  science  of  the  application  of  knowledge  to  practical 
purposes”  {Wehster^s  Medical  Desk  Dictionary,  1986).  Pharmaceutical  manufacturers  use  scientific 
knowledge  to  develop  and  produce  drugs  whose  purpose  is  to  improve  health  and  well-being.  Thus, 
new  pharmaceuticals  are  new  technologies,  and  our  interest  in  improving  a  managed  care 
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organization’s  ability  to  review  and  adopt  new  pharmaceuticals  is  driven  by  the  assumption  that 
informed  decisionmaking  on  new  technologies  will  lead  to  superior  outcomes. 

Informing  decisionmaking  is  the  goal  of  technology  assessment,  which  draws  upon  a  wide 
variety  of  analytic  tools  and  methods  to  take  a  comprehensive  approach  to  evaluation.  Although  it 
was  initially  intended  to  be  a  mostly  centrahzed  activity,  a  variety  of  organizations  now  use 
technology  assessment  to  assist  decisionmaking  on  technologies  from  their  own  perspective  rather 
than  a  regional  or  national  one  (Goodman,  1998).  Technology  assessment  in  health  care  tends  to  be 
performed  in  response  to  the  introduction  of,  or  major  shift  in  the  demand  for,  a  new  technology.  It 
has  been  defined  as  the  “evaluation  of  the  safety,  effectiveness,  and  appropriateness  of  the  many 
devices,  medical  and  surgical  procedures,  and  pharmaceuticals  promoted  to  improve  a  patient’s 
condition  or  quality  of  life”  (Matuszewski,  1997).  This  report  presents  a  narrower  focus  on 
pharmaceutical  technology  assessment  to  support  decisionmaking  for  populations  enrolled  in 
managed  care."^ 

For  adoption  of  new  pharmaceuticals,  pharmaceutical  technology  assessment  should 
address  a  range  of  issues,  including  safety,  clinical  effectiveness,  and  cost-effectiveness  in  terms  of 
medical  expenditures.  Depending  on  the  particular  condition  and  the  perspective  of  the  evaluator, 
issues  may  also  include  comphcating  factors  such  as  indirect  costs,  labor-market  outcomes  (e.g., 
absenteeism,  work-related  disability  and  injury,  and  employee  turnover),  ethics,  belief  systems, 
public  pohcy  decisions,  and  impact  on  provider  and  patient  roles.  Finally,  the  market  or  pubhc 
relations  impHcations  of  decisions  can  be  significant.  For  example,  patient  protests  made  a  decision 
to  exclude  Prozac  from  a  large  health  plan’s  formulary^  into  headHne  news. 

This  report  is  written  from  the  perspective  of  payers,  providers,  and  society,  rather  than 
from  that  of  the  individual  patient.  We  refer  to  health  plans  throughout  this  document  because 
their  breadth  of  responsibility  in  population-based  medical  decisionmaking  encompasses  clinical 
and  financial  outcomes.  However,  this  discussion  should  also  be  applicable  to  integrated  and 
hospital-based  provider  organizations,  multi-specialty  groups,  and  independent  practice 
associations. 

Organization  of  This  Report 

This  report  focuses  on  how  health  plans  and  provider  organizations  operating  in  managed 
care  environments  (“managed  care  organizations”)  do  and  could  evaluate  the  opportunities  and 


4  To  learn  more  generally  and  extensively  about  health  care  technology  assessment,  refer  to  Goodman’s 
excellent  monograph  (1998). 

5  A  formulary  is  a  list  of  covered  or  available  drugs,  often  accompanied  by  a  set  of  rules  or  guidelines  for 
their  use. 
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challenges  presented  by  new  pharmaceuticals.  We  discuss  how  organizational  decisionmakers  can 
use  the  tools  of  technology  assessment  to  help  estimate  the  likely  effect  of  an  emerging 
pharmaceutical  technology,  and  their  response  to  that  technology. 

We  intend  this  report  to  assist  managed  care  organizations  interested  in  initiating  formal 
pharmaceutical  evaluation  activities  or  in  improving  existing  procedures.  In  this  study,  we  identify 
gaps  in  the  current  practices  of  pharmaceutical  technology  assessment  and  propose  improvements. 
We  first  employed  a  two-part  data-coUection  process  of  interviews  and  a  literature  review,  described 
in  the  first  part  of  Section  2.  We  then  synthesized  the  data  into  a  review  of  current  practices, 
presented  in  the  second  part  of  Section  2.  This  review  informed  Section  3,  which  contains  both  a 
prescriptive  framework  for  futxire  pharmaceutical  technology  assessment  and  a  guide  to  the  major 
relevant  analytic  approaches.  Section  4  discusses  practical  aspects  of  applying  the  framework. 
Appendix  A  provides  an  annotated  bibliography;  Appendix  B  contains  the  interview  questionnaire. 
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2.  A  Review  of  Current  Practices 

For  oxir  review  of  current  practices,  we  collected  data  through  interviews  and  a  hterature 
review.  We  structured  the  interviews  and  literature  review  around  five  questions  that  are  central  to 
understanding  the  nature  of  current  assessments  for  new  pharmaceuticals,  and  how  they  translate 
into  benefit  language  and  individual  coverage  decisions: 

•  Who  decides  to  accept  or  reject  new  pharmaceutical  technology,  and  how? 

•  What  decision  options  (e.g.,  “Exclude  drug,”  “Assign  to  high-cost  tier,” and  '“Require prior 
authorization”)  are  considered? 

•  What  factors  affect  the  timing  and  resources  devoted  to  pharmaceutical  technology  assessment? 

•  What  types  of  evaluations  are  performed  and  what  types  of  evidence  do  they  use? 

•  How  are  decisions  implemented  and  the  results  monitored? 


Prior  to  our  data  collection,  we  hypothesized  relationships  between  organizational 
motivations  for  assessing  a  new  technology  and  who,  what,  how,  and  when  pharmaceutical 
technology  assessments  were  conducted.  For  example,  consider  a  preferred  provider  organization 
(PPO)  with  indemnity  prescription  drug  coverage  for  aU  FDA-approved  drugs.  Such  a  provider  woxild 
likely  conclude  that  it  only  needs  to  determine  whether  a  new  drug  is  specifically  excluded  fi*om 
coverage  because  of  plan  contract  language  (e.g.,  is  considered  a  cosmetic  or  in  another  excluded-drug 
class).  We  would  expect  that  such  an  assessment  would  be  done  with  minimum  effort  and  analysis, 
and  that  it  would  likely  wait  until  after  the  drug  becomes  available.  In  contrast,  an  HMO  with  fuU 
financial  risk  for  drug-benefit  costs  (i.e.,  risk  contract),  a  tightly  managed  formvilary,  and  an 
extensive  authorization  program  would  likely  see  itseK  as  having  much  broader  and  more  clinically 
detailed  assessment  needs.  We  would  expect  that  this  organization’s  assessment  process  would 
begin  much  earlier,  perhaps  even  as  a  result  of  monitoring  the  progress  of  pre-market  drugs;  that  it 
would  probably  take  longer;  and  that  more  resources  and  more  techniques  wo\ild  be  brought  to  bear 
on  the  assessment. 

In  this  section,  we  first  describe  the  data  collection  that  we  designed  to  address  these 
hypothesized  relationships,  then  synthesize  the  findings. 

Data  Collection 

We  collected  data  about  current  practices  for  this  study  in  two  ways:  (1)  by  quahtatively 
interviewing  a  purposive  (i.e.,  nonrandom)  sample  of  relevant  organizations,  and  (2)  by  searching 
the  pertinent  hterature  on  existing  and  recommended  practices  in  health  care  technology 
assessment. 
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interviews 

We  designed  the  semi-structured  qualitative  interviews  to  review  pharmaceutical  technology 
assessment  goals  and  practices  at  a  variety  of  managed  care  organizations.  Budget  constraints 
prevented  us  from  using  probability  sampling  to  select  the  organizations.  Rather,  we  chose  a 
purposive  sample  of  organizations  that  reflected  the  target  population  in  geographic  dispersion, 
position  in  the  market,  and  activity/risk  related  to  pharmaceuticals.  After  making  our  selections, 
we  spoke  with  the  medical  director  or  lead  pharmacy  contact  at  three  of  three  selected  national 
managed  care  plans,  three  of  three  regional  managed  care  plans,  four  of  four  medical  groups 
(differentiated  by  structure  under  which  they  operate:  staff,  multi-specialty,  IPA  or  PPO,  etc.),  and 
two  of  three  selected  pharmacy  benefit  management  (PBM)  firms. ^  Our  interviewees  at  these 
organizations  were  senior  managers  with  responsibihty  for  financial  Hability  and  clinical  quality  in 
their  organizations.  These  managers  were  also  typically  responsible  for  pharmaceutical-related 
administrative,  legal,  and  regulatory  compliance.  Detailed  notes  from  each  of  these  interviews  were 
taken  for  internal  use;  for  reasons  of  confidentiality,  they  are  not  included  in  this  report. 

The  12  interviews  were  conducted  over  the  telephone  by  one  of  three  interviewers  with 
extensive  managed  care  experience,  usually  with  a  supporting  investigator  listening  in  as  a  scribe. 
Interviews  lasted  from  30  to  60  minutes.  The  interview  questionnaire,  which  we  faxed  to 
interviewees  in  advance,  is  included  in  Appendix  B. 

Literature  Review 

For  the  literature  review,  we  adapted  the  methods  of  the  Southern  California  Evidence- 
Based  Practice  Center  (The  Cochrane  Library,  1999)  to  identify  and  summarize  articles  containing 
information  on  current  technology  assessment  practices,  including  specific  assessment  techniques 
and  suggestions  for  future  practice.  First,  we  conducted  four  separate  literature  database  searches 
to  ensure  that  our  search  was  comprehensive.  With  the  assistance  of  a  professional  Hbrarian,  we 
chose  keywords  and  other  search  parameters  in  an  iterative  manner,  using  articles  we  had 
prospectively  identified  as  important.  These  four  searches  located  57,  115,  159,  and  765  articles; 
thus,  the  unduplicated  hsts  contained  roughly  1,000  articles.  As  well,  experts,  including  colleagues 
in  the  pharmaceutical  industry,  suggested  articles  that  were  important  to  include  in  our  review.  In 
addition  to  the  Hbrary  searches,  we  obtained  those  articles. 


^  Pharmaceutical  Benefits  Managers  (PBMs)  offer  drug  benefit  packages  to  managed  care  organizations, 
employer  groups,  and  other  payers.  These  organizations  originally  provided  claims-processing  services  for 
prescription  drugs,  but  they  also  negotiate  prescription  prices  with  pharmacies,  implement  generic 
substitute  programs,  and  are  developing  volume-purchase  agreements  with  pharmaceutical  manufacturers 
(Navarro,  1994). 
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Next,  we  screened  each  article  title  for  relevance,  using  a  pair  of  readers  (a  health  services 
researcher  and  a  physician)  selected  from  among  three  project  members.  We  subsequently  obtained 
the  abstract  for  articles  deemed  relevant  by  either  of  the  two  screeners,  which  reduced  the  initial 
pool  of  articles  to  126  candidates  for  further  review.  We  screened  the  abstracts  of  these  126  articles 
and  identified  38  general-overview  articles  describing  current  practices  or  recommended  future 
directions.  We  obtained  the  full  text  of  these  38  articles. 

One  project  member  then  scanned  each  of  the  38  articles  and  chose  some  key  articles  for 
immediate  reading  by  the  project  team.  We  used  the  material  contained  in  these  articles  to  identify 
the  technology  assessment  methodologies  and  other  subtopics  that  we  wished  to  include  in  this 
report.  We  then  returned  to  the  original  search  fists  and,  using  a  screening  process  similar  to  that 
described  above,  selected  articles  relevant  to  each  subtopic.  Additional  project  members  read  and 
summarized  the  subtopic  and  remaining  38  overview  articles.  Because  these  references  constitute 
an  important  source  of  material  for  the  text,  we  have  summarized  them  in  the  annotated 
bibliography.  Appendix  A. 

More  comprehensive  approaches,  such  as  systematic  searching  for  books  and  non-English 
literature  (see  the  Evidence  Assessment  portion  of  Table  4.1  for  elements  of  a  thorough  review),  were 
not  possible  because  of  time  and  budget  constraints.  However,  the  search  methods  that  we 
employed  were  systematic  and  reproducible,  and  we  believe  that  we  have  captured  most  of  the 
relevant  literature. 

Current  Approaches 

Health  care  organizations  use  a  wide  variety  of  strategies  to  assess  pharmaceutical 
technology  and  manage  pharmacy  benefits.  The  health  plans,  PBMs,  integrated  health  systems, 
and  medical  groups/IPAs  we  interviewed  all  devote  resources  to  reviewing  new  drugs.  However,  the 
purpose,  timing,  formality,  rigor,  and  criteria  for  review  varied  widely,  as  did  the  resources  allocated 
to  the  process  and,  relatedly,  the  range  of  possible  responses  available  to  the  reviewers.  The 
literature  documents  a  similar  variation  in  the  technology  assessment  process  and  resulting 
coverage  decisions.  The  following  subsections  synthesize  the  findings  of  interviews  and  literature 
review,  by  framing  question. 

Who  Decides  to  Accept  or  Reject  New  Pharmaceutical  Technology,  and  How? 

Many  organizations  and  individuals  make  decisions  that  affect  prescription  drug  benefits 
and  adoption  of  new  drug  technology.  Health  plans  decide  which  benefit  plans  they  will  offer  to 
payers,  including  whether  and  under  what  conditions  they  will  offer  a  prescription-drug  benefit  or 
cover  a  particular  drug.  Employers  and  other  health  plan  sponsors  decide  whether  and  what  kind  of 
prescription-drug  benefit  they  wifi  purchase.  PBMs  usually  recommend  a  range  of  prescription-drug 
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benefit  packages  for  their  chents,  but  they  will  also  administer  their  clients’  own  unique  benefit 
plans.  Physician  groups  and  hospital  systems  decide  which  drugs  they  will  use  preferentially  in 
their  practices.  Finally,  patients  and  their  famihes  decide  when  to  request  specific  drugs,  whether 
they  are  willing  to  accept  having  to  make  higher  payments  to  get  a  specific  drug,  and  when  to 
complain  about  access  to  a  specific  drug.  Although  all  of  these  actors  influence  pharmaceutical 
decisionmaking,  the  remainder  of  this  section  focuses  specifically  on  health  plan  and  medical  group 
decisionmakers. 

Our  interviews  indicated  that  the  locus  of  pharmaceutical  evaluation  varies  by 
organizational  type.  National  or  multi-regional  health  plans  reported  that  core  benefit-design 
decisions  and  the  assessment  of  new  and  emerging  technologies  are  typically  made  at  the  corporate- 
office  level.  However,  their  local  subsidiaries  generally  make  patient-specific  decisions  about  the 
medical  necessity  of  specific  drugs  and  physician-specific  decisions  regarding  compliance  with  clinical 
use  guidelines.  Local  subsidiaries  also  have  some  flexibility  in  authorizing  which  benefit  plans  they 
offer.  Moreover,  all  of  the  regional  and  national  plans  we  interviewed  indicated  that  their  local 
subsidiaries  were  represented  in  some  way  on  their  pharmaceutical  and  therapeutics  (P  &  T) 
committees  and/or  health  plan  coverage  committees  at  the  organization’s  corporate  level. 

Pharmaceutical-coverage  decisions  can  be  made  at  several  different  levels  within  a  health 
plan.  All  of  the  organizations  we  interviewed  had  at  least  three  possible  levels  for  the  review  and 
^PP^^oval  of  new  pharmaceutical  technology  or  new  coverage  or  control  decisions:  some  form  of  a  P  & 
T  committee  (variously  named),  a  medical  director  and/or  a  pharmacy  director,  and  a  senior 
management  group  that  reviewed  high  cost,  controversial  or  groundbreaking  assessment,  coverage, 
or  control  decisions.  Some  health  plans  also  had  a  separate  structure  for  developing  and  carrying 
out  specific  implementation  decisions,  such  as  whether  to  impose  prior-authorization  requirements 
or  develop  and  implement  clinical  guidehnes. 

Differences  in  financial  incentives  and  potential  financial  impact  at  the  organizations  that 
we  interviewed  affected  which  organizational  level  decides  to  accept  or  reject  new  pharmaceuticals. 
In  general,  new  drugs  or  indications  in  existing  therapeutic  classes  are  reviewed  and  approved  at 
the  P  &  T  committee  and  medical-director  level,  unless  they  entail  major  new  costs  or  expansion  of 
benefits,  or  raise  sensitive  issues  regarding  demand  and  use  (e.g.,  abortifacients,  agents  with  both 
cosmetic  and  therapeutic  uses).  Decisions  that  are  generally  referred  to  a  more  senior  level  involve 
potential  blockbuster  drugs, drugs  that  create  a  new  and  potentially  expensive  therapeutic  class, 
and  drugs  on  which  decisions  may  have  particular  legal  or  marketplace  sensitivity  (e.g.,  Viagra,  Cox 


A  blockbuster  drug  is  one  that  is  unusually  expensive,  has  a  major  effect  on  cost  and/or  quality,  and/or 
sets  a  precedent  for  future  coverage  or  practice. 
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2  inhibitors,  anti-HIV  drugs).  Decisions  to  change  benefit-plan  language  t5rpicaUy  go  through  an 
additional  process  of  review  and  approval. 

Our  hterature  review  revealed  a  similar  picture.  Steiner  et  al.  (1996)  found  that  small 
health  plans  and  indemnity  plans  were  generally  more  likely  to  rely  exclusively  on  medical  directors 
for  technology  assessment.®  Lyles,  Luce,  and  Rentz  (1997)  found  that  96%  of  health  plans  utilize  a 
formulary  committee  to  assess  new  or  innovative  pharmaceuticals,  92%  said  their  medical  directors 
are  directly  involved  with  the  review  process,  and  46%  indicated  that  the  medical  director 
determines  whether  or  not  to  pursue  further  analysis.  However,  these  authors  also  found  that 
group-  and  staff-model  health  plans,  as  well  as  PBMs,  do  not  usually  involve  the  medical  directors 
in  formulary  decisions.  This  is  a  point  of  contrast  with  our  interviews,  which  found  that  the  medical 
directors  usually  play  a  significant  role  in  decisions  affecting  prescription  practice  management, 
control  mechanisms,  and  the  timing  of  drug  assessments,  even  if  the  organization  designates  others 
to  manage  the  day-to-day  process  of  assessing  pharmaceutical  approvals  of  use. 

Our  interviews  indicated  that  who  conducts  the  new-drug  assessments  is  a  function  of  the 
resources  devoted  to  such  assessments.  Small  plans  and  medical  groups  often  rehed  on  more- 
informal  decisions  by  the  medical  director  or  pharmacy  director  to  determine  which  drugs  to  assess 
fully.  These  findings  are  consistent  with  Steiner  et  al.  (1996),  who  found  that  smaller  health  plans 
were  twice  as  hkely  as  larger  plans  to  indicate  that  the  medical  director  alone  either  reviews  the 
new  technology  or  has  discretion  to  involve  others  in  the  review  process.  We  interviewed  one  local 
plan  with  a  closed  formulary  but  an  active  process  to  authorize  prescriptions  for  nonformulary  drugs. 
That  plan  reported  that  they  generally  wait  untH  a  drug  has  been  in  use  in  their  local  marketplace 
for  a  year  before  considering  it  for  formulary  inclusion.  This  delayed  approach  allows  their  small 
assessment  staff  to  focus  on  current  priorities  and,  perhaps  more  significantly,  to  depend  on 
assessments  from  other  organizations  to  assist  their  review. 

All  of  the  organizations  had  the  responsible  individuals  summarize  the  results  of  their 
assessments  in  an  internal  report,  although  each  organization  had  its  own  name  and  format  for  this 
report.  These  reports  usually  include  a  description  of  the  drug  (generic  name  and  brand  name, 
therapeutic  class,  mechanism  of  action,  etc.),  a  discussion  of  its  safety  and  efficacy,  and  estimates  of 
its  acquisition  and,  sometimes,  administration  cost.  When  alternatives  are  available,  especially  if 
they  are  already  covered  or  preferred  by  the  organization,  the  reports  generally  include  a  discussion 
of  the  comparative  safety,  efficacy,  and  costs  of  these  alternatives.  Some  organizations  summarize 
their  assessments  in  a  table  displaying  the  costs,  efficacy,  and  safety  for  the  drug  alone  and  in 


®  Steiner  also  found  that  indemnity  plans,  which  we  did  not  survey,  were  three  times  more  likely  than 
HMOs  to  indicate  that  the  medical  director  alone  should  be  responsible  for  the  final  medical-coverage 
decision. 
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comparison  with  existing  available  therapies.  No  organization  reported  being  satisfied  with  its 
ability  to  quantify  cost-effectiveness. 

In  most  organizations  we  spoke  to,  a  pharmacist  (e.g.,  the  pharmacy  director)  was 
responsible  for  the  reports,  although  a  medical  director  was  involved  in  all  cases  and  consulting 
specialists  in  many.  The  reports  tended  to  advocate  a  particular  decision  or  recommend  further 
study  or  review.  In  all  but  the  smaller  medical  groups,  additional  committees  and  management 
teams  had  some  responsibihty  for  some  of  the  decisions  affecting  drug  coverage  and  control  policy,  so 
that  recommendations  go  to  management  for  approval  and,  if  appropriate,  to  others  for  further 
review  or  implementation  planning. 

Most  organizations  also  employed  a  separate  committee  or  medical  management  structure 
to  evaluate  recommendations  to  change  drug-control  mechanisms,  operating  programs,  provider 
contracts,  or  provider  education.  Those  with  formal  disease-management  programs  uniformly 
described  a  separate  management  and  decision  structure  for  these  programs  that  could  choose  to 
coordinate  with  the  P  &  T  committee  or  make  separate  and  different  decisions.  Finally,  each 
interviewed  health  plan  had  a  separate  technology  assessment  committee  responsible  for  reviewing 
procedures,  devices,  and  other  non-pharmaceutical  technology.  In  some  cases,  though,  approving  a 
procedure  or  device  may  involve  making  a  decision  to  cover  some  drug  therapy  associated  with  that 
procedure.  In  these  instances,  the  technology  assessment  committee  may  either  coordinate  with  the 
P  &  T  committee  or  assume  responsibility  over  that  specific  decision  (e.g.,  localized  chemotherapy 
infusion).  The  latter  avenue  is  common,  because  each  of  the  committees  and  staff  groups  mentioned 
may  report  to  separate  parts  of  the  organization,  complicating  the  formation  of  coherent  approaches. 

Although  every  interviewee  could  explain  the  general  logic  of  who  made  what  decisions 
regarding  new  pharmaceutical  technology  for  his  or  her  organization,  the  assessment  criteria  that 
were  used  varied  on  a  case-by-case  basis.  All  the  health  plans  we  interviewed  agreed  with  the 
hterature  that  a  “final  decision  (on  a  technology)  has  to  consider  the  cost  to  pohcyholders,  the  cost  for 
implementing  coverage,  and  business  imphcations  such  as  competition”  (Luce  and  Brown,  1995). 

The  larger  the  organization,  the  more  likely  decisionmaking  criteria  were  formahzed  into  policies 
and  procedures;  however,  even  in  large  organizations,  the  definition  of  which  decisions  should  be 
referred  for  additional  review  or  planning  appeared  to  be  subjective  and  subject  to  management 
discretion.  Recommendations  requiring  changes  to  benefit-plan  language  were  an  exception;  all 
organizations  had  a  separate  committee  or  senior  management  group  review  such  actions. 

What  Decision  Options  Are  Considered? 

Choosing  the  best  possible  pohcy  requires  a  clear  understanding  of  what  decision  options  are 
available  without  (but,  on  occasion,  after)  renegotiating  contracts  with  chents,  PBMs,  providers,  or 


-  12- 


Pharmaceutical  Technology  Assessment  for  Managed  Care 


manufacturers.  Our  interviews  revealed  that  each  type  of  organization  had  a  different  range  of 
options  to  consider  based  on  its  role  and  its  prescription-drug  benefit  structure,  management  control 
mechanisms,  and  contractual  relationships.  The  staff-model  HMOs  and  medical  groups  with  in- 
house  pharmacies  that  we  interviewed  reported  they  could  easily  implement  closed  formularies  and 
pharmacist-driven  utiHzation  controls,  whereas  such  controls  would  be  more  difficult  in  a  loosely 
structured  IPA-model  HMO  or  PPO  plan.  Medical  groups  indicated  that  their  inabihty  to  define  the 
pharmaceutical-benefit  plans  or  the  management-control  mechanisms  of  health  plans  with  which 
they  contract  greatly  limits  their  decision  options.  However,  some  do  construct  lists  of  preferred 
injectables  and  other  drugs  to  be  used  in  their  offices,  or  an  internal  fists  of  preferred  “take-at-home” 
prescription  drugs  from  the  formularies  provided  by  the  health  plans,  sometimes  coupled  with  an 
appeal  for  an  expansion  of  or  an  individual  exception  to  the  formulary.  Some  medical  groups  also 
choose  how  to  educate,  monitor,  and  enforce  drug  policies  and  preferences  within  their  organization. 
Some  of  the  medical  groups  that  we  interviewed  concentrate  on  drug  classes  with  high  utilization 
rates  and  costs,  rather  than  on  all  therapeutic  classes.  Of  note,  these  groups  were  at  some  financial 
risk  for  drug  benefit  costs,  so  it  is  not  surprising  that  their  fists  focused  on  pharmaceuticals  for 
which  the  group  had  the  potential  to  divert  prescribing  habits  to  more  cost-effective  alternatives. 

Health  plans,  in  contrast,  appear  to  have  a  broad  range  of  decision  options  for  actions 
regarding  pharmaceuticals.  The  more  common  options  identified  in  our  interviews  fall  into  one  of 
three  categories:  actions  relating  to  evaluation,  to  altering  drug-benefit  design  features,  and  to 
implementing  management  control  strategies.  We  list  some  commonly  mentioned  options  in  all 
three  categories  in  Tables  2.1  through  2.3.  These  options  are  related,  since  the  evaluation  options 
available  to  a  health  plan  depend,  in  part,  on  the  decisions  it  makes  regarding  plan  design  features 
and  management  control  tactics. 

Not  all  the  plans  mentioned  all  the  options.  Entire  groups  of  decision  options  were 
imavailable  or  irrelevant  to  some  plans  or  PBMs.  In  some  cases,  this  unavailability  related  to  the 
design  of  their  prescription  drug  benefits  and  the  infrastructure  available  to  support  the 
management  of  pharmaceuticals.  Organizations  did  not  need  to  consider  any  of  the  decision  options 
related  to  closed  formularies,  such  as  step  programs  or  prior  authorization  of  nonformulary  drugs,  if 
they  had  open  formularies.^  Organizations  did  not  develop  detailed  clinical  practice  guidelines,  local 
counter-detailing  programs,  or  provider-directed  disease-management  programs  unless  they  had  the 
provider  contractual  relationships  and  the  local  infrastructxrre  necessary  to  do  utilization 
review/management,  perform  on-site  physician  education,  or  directly  influence  clinical  behavior. 


9  This  group  includes  many  organizations.  Lyles,  Luce,  and  Rentz  (1997)  found  that  41%  of  the  large 
health  plans  in  his  survey  had  open  formularies,  and  Schulman  et  al.  (1996)  found  that  PBMs  use  an 
open  formulary  plan  structure  for  the  majority  of  their  employer  and  HMO  clients. 
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Organizations  also  rejected  some  decision  options  that  they  considered  to  be  outside  of  an 
acceptable  range  of  management  control  tactics;  that  is,  options  that  they  consider  inconsistent  with 
their  benefit  design  and  management-control  philosophy. 

The  hterature  indicates  that  organizations  may  ehminate  large  groups  of  decision  options 
because  of  the  costs  involved  in  implementation.  For  example,  Garber  (1994)  asserted  that  making 
indication-specific  coverage  decisions  requires  a  costly  administrative  effort.  Consistent  with  that 
assertion,  the  organizations  we  surveyed  were  quite  cautious  about  creating  special  categories  for 
prior  authorization  or  limited  coverage.  In  fact,  most  organizations  that  we  contacted  did  not  have 
prior-authorization  or  exception-processing  mechanisms,  and  others  used  them  rarely  (for  less  than 
5  to  10%  of  prescriptions).  Some  executives  also  expressed  concern  that  special  co-payment  rates 
and  unique  rules  for  specific  drugs  make  their  benefit  plan  appear  complicated,  and  thus  difficult  to 
market. 

Some  organizations  appear  to  be  simply  unaware  of  some  viable  options.  However,  all  the 
organizations  we  interviewed  expressed  interest  in  learning  more  about  the  full  range  of  available 
options.  All  also  indicated  that  which  pharmaceutical  technology  assessment  they  would  need 
would  depend  on  the  specific  decision  options  available  to  their  organization,  and  the  relative  costs 
and  returns  to  the  options. 
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What  Factors  Ajfect  the  Timing  and  Resources  Devoted  to  Pharmaceutical  Technology 
Assessment? 

None  of  the  organizations  we  interviewed  reported  using  a  formal  analysis  of  the  possible 
impact  of  every  new  drug  or  indication,  such  as  that  described  by  Eddy  (1989),  to  decide  which  drugs 
to  subject  to  a  more  thorough  assessment  process.  Rather,  we  found  that  an  organization's  overall 
mission  and  market  strategy  may  determine  what  pharmaceuticals  are  assessed,  which  is 
consistent  with  Luce  and  Brown's  findings  (1965)  about  the  motivations  for  medical  technology 
assessment  in  general.  Also  similar  to  Luce  and  Brown  (1995),  organizations  that  we  interviewed 
conducted  assessments  in  order  to  variously  improve  care,  contain  costs,  make  prudent  purchasing 
decisions,  support  chnical  coverage  decisions,  determine  experimental  status,  and  comply  with 
federal  regulations.  In  this  context,  we  found  that  an  organization's  reasons  for  performing  an 
assessment  of  a  new  pharmaceutical  technology  seems  to  substantially  determine  the  timing, 
resources,  and  rigor  devoted  to  it. 

The  timing  of  pharmaceutical  technology  assessments  varied  widely  among  the 
organizations  that  we  interviewed.  Larger  health  plans,  particularly  those  with  full  financial  risk 
for  pharmaceutical  use,  but  generally  not  PBMs  or  provider  organizations,  specifically  tasked 
individuals  with  scanning  the  drug-development  pipeline  (trade  papers,  journals,  etc.)  to  ensure  that 
the  organization  is  aware  of  any  potential  blockbuster  drugs,  new  therapeutic  classes,  or  other 
pharmaceutical  innovations  that  might  have  a  major  impact  on  health  plan  costs  or  outcomes.  This 
scanning  activity  consisted  of  a  review  of  industry  and  governmental  pubHcations,  and  often  involved 
formal  and  informal  communications  and  meetings  with  drug  manufactxirers.  Similarly,  Luce  and 
Brown’s  (1995)  observed  that  larger  plans  often  begin  their  assessments  before  Food  and  Drug 
Administration  (FDA)  approval,  although  many  conduct  full  assessments  on  only  a  hmited  subset  of 
new  pharmaceuticals. 

Decisionmakers  for  these  organizations  have  incentives  to  perform  early  assessment  that 
appear  to  go  beyond  immediate  cost.  Health  plans  and  health  care  systems,  which  must  set  annual 
or  multi-year  premiums  many  months  before  a  benefit  plan  year  begins,  need  to  anticipate  whether 
any  new  drugs  in  the  development  pipehne  will  substantially  raise  or  lower  their  costs.  Likewise, 
plans  with  formularies  that  rely  on  extensive  chnical  guidelines  for  prior  authorization  or  step 
programs  need  to  begin  their  assessment  of  important  new  drugs  early,  so  that  their  findings  can  be 
incorporated  into  the  plan's  benefit  language  and  management-control  processes. 

Our  interviews  with  PBMs  supported  Luce  and  Brown’s  observation  that  PBMs,  which 
assume  httle  or  no  financial  risk  for  benefit  costs,  typically  conduct  their  full  assessments  after  a 
drug  receives  FDA  approval;  and  Schulman's  (1996)  assertion  that  the  PBMs  generally  wait  until 
about  12  months  of  chnical  experience  before  formally  considering  a  drug  for  inclusion  on  a  preferred 
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formulary  list.  Subjects  from  all  types  of  organizations  complained  that,  around  the  time  of 
approval,  information  may  be  limited  to  that  provided  by  manufacturers  who  are  approaching 
health  plans  and  PBMs  to  facilitate  coverage  irrespective  of  the  organization's  own  timeline  for 
pharmaceutical  technology  assessments. 

The  resources  devoted  to  assessing  new  pharmaceutical  technology  vary  significantly  among 
organizations.  According  to  our  interviewees,  the  personnel  resources  devoted  to  pharmaceutical 
technology  assessment  ranged  from  a  staff  of  30  and  multiple  levels  of  managers  and  paid 
committees,  to  a  single  director  of  pharmacy  with  a  staff  of  one  to  two  persons  and  a  volunteer  P  & 

T  committee.  In  contrast,  only  75%  of  the  51  plans  interviewed  by  Lyles,  Luce,  and  Rentz  (1997) 
reported  having  any  employee  responsible  for  pharmaceutical  assessment.  This  discrepancy  may  be 
partially  explained  by  the  fact  that  all  of  our  interviewees  relied  upon  internal  assessments, 
whereas  only  74%  of  the  plans  in  the  Lyles,  Luce,  and  Rentz  study  did. 

To  some  extent,  these  differences  in  resources  reflect  the  differences  in  the  sizes  of  the 
organizations  and  annual  pharmacy  expenditures.  They  also  reflect  differences  in  the  purposes  of 
the  assessments.  In  staff-model  HMOs  or  large  medical  groups  with  in-house  or  hospital 
pharmacies,  pharmaceutical  technology  assessment  can  serve  broad  functions,  such  as  budget 
prioritization,  improved  strategic  planning,  alleviating  pressure  from  third-party  payers,  improving 
quahty,  and  avoiding  obsolescence  (Luce  and  Brown,  1995).  In  fact,  early  adoption  of  technology  in 
some  organizations  may  be  part  of  the  marketing  strategy.  However,  each  of  these  purposes 
requires  differing  levels  of  staff  support  and  other  resources. 

The  type  and  rigor  of  an  assessment  necessarily  affect  the  resources  that  an  organization 
devotes  to  it.  For  example,  a  health  plan  whose  drug  benefits  cover  all  FDA- approved  prescription 
drugs  without  regard  to  indication,  unless  specifically  excluded  by  the  plan,  would  need  to  devote  few 
resources  to  pharmaceutical  assessment.  The  only  decision  options  such  organizations  would  have 
to  consider  are  whether  the  drug  in  question  is  excluded  by  the  plan  and  whether  the  existence  of 
the  new  drug  or  indication  requires  or  suggests  modifying  the  health  plan's  benefit  language.  It  is 
important  to  note  that  the  latter  may  not  be  apparent  until  some  evaluation  is  performed.  For 
example,  investigation  of  an  agent  that  is  superficially  uninteresting  (e.g.,  expensive,  excluded,  or 
not  unique  or  decisive)  may  suggest  that  the  drug  could  substitute  for  those  more  expensive  or  less- 
effective  services  that  the  plan  is  hable  for,  and  that  pohcies  that  promote  the  use  of  the  drug  are 
desirable. 

The  organizations  that  we  interviewed  most  commonly  perform  chnical-effectiveness  and 
safety  assessments,  followed  by  cost-of-treatment,  cost-effectiveness,  and,  infrequently,  quality-of- 
hfe  assessments  (Lyles,  Luce,  and  Rentz,  1997).  Our  interviews  are  again  consistent  with  the 
findings  of  Lyles,  Luce,  and  Rentz  (1997)  and  Luce  and  Brown  (1995),  who  reported  that  cost- 
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effectiveness  and  quality-of-life  analyses  are  limited  by  the  lack  of  directly  comparative  data 
between  competitor  drugs  in  real-world  settings.  Therefore,  if  an  organization  chooses  to  prioritize 
one  drug  over  others  in  the  same  class,  it  will  need  to  devote  extra  resources  to  filhng  information 
gaps  on  comparative  or  incremental  cost-effectiveness.  On  the  other  hand,  if  such  an  organization  is 
considering  covering  a  promising  new  drug  in  a  new  therapeutic  class,  it  might  need  to  review  only 
safety  and  efficacy  data;  cost-effectiveness  data  may  be  less  relevant. 

All  of  the  organizations  that  we  interviewed  used  some  externally  developed  assessment 
material  in  their  internal  processes.  In  addition,  aU  of  the  PBMs  we  sxirveyed  considered  new-drug 
assessment  to  be  a  service  they  can  provide  to  their  clients,  either  as  part  of  a  package  of  pre¬ 
determined  formularies  and  control  strategies,  or  as  a  separate  product.  Many  other  organizations 
also  provide  assessments  of  medical  technology,  including  procedures,  devices,  and  pharmaceuticals. 
(A  list  of  sources  of  public  and  proprietary  pharmaceutical  assessments  can  be  found  in  The 
Directory  of  Health  Technology  Assessment  Organizations  Worldwide,  1999.)  The  availability  of  these 
materials  did  not  prevent  organizations  from  conducting  their  own  internal  assessments  on  the 
drugs  that  they  considered  important,  possibly  because  of  these  materials’  objective  shortcomings 
but  also  because  of  an  expressed  distrust  of  some  external  assessment  information 

Wtat  Types  of  Evaluations  Are  Performed  and  What  Types  of  Evidence  Do  They  Use? 

We  found  that,  even  within  a  single  organization,  varying  motives  lead  to  different  processes 
for  reviewing  different  drugs.  For  example,  some  interviewees  reported  that  health  plans  paying 
physicians  or  hospitals  a  flat  fee,  a  per  diem  rate,  or  capitation  do  not  devote  the  same  resources  to 
reviewing  drugs  used  solely  in  the  office  or  inpatient  setting  (i.e.,  included  in  the  flat  rate)  that  they 
would  to  outpatient  prescription  drugs  for  which  they  are  financially  and  administratively 
responsible.  Likewise,  physician  groups  and  hospitals  did  not  report  spending  resources  on 
reviewing  drugs  that  are  mandated  for  coverage  by  their  contracting  health  plans,  except  when  they 
expected  clear  differences  in  clinical  outcomes  between  drugs  in  the  same  class  or  differences  in  cost 
that  could  significantly  impact  the  organization’s  finances. 

Steiner  et  al.  (1996)  documented  the  degree  to  which  technology  assessments  vary  from  plan 
to  plan,  that  the  content  of  the  technology  assessment  performed  by  the  plans  was  difficult  to 
predict,  and  that  plan-conducted  assessments  differ  from  assessments  performed  by  the  scientific 
commiinity.  They  note  that  health  plans  confer  varying  levels  of  authority  on  their  medical  directors, 
differ  in  the  timing  of  their  technology  assessment  decisions,  and  view  sources  and  evidence 
differently.  Steiner  et  al.  (1997)  also  demonstrated  that  these  variations  in  technology  assessment 
resulted  in  substantial  variation  in  coverage  for  emerging  medical  device  technology.  Luce  and 
Brown  (1995)  found  that  hospitals,  HMOs,  and  other  types  of  insurers  all  conduct  technology 

assessments,  but  that  the  form  and  sophistication  of  their  analyses  range  widely.  Weingart’s 
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interviews  (1993)  of  12  Academic  Medical  Center  Consortium  institutions  produced  similar 
observations:  variation  in  the  technology  assessment  process,  lack  of  explicit  evaluation  criteria,  and 
the  presence  of  political,  informal,  or  ad  hoc  decisions. 

Despite  the  wide  variation  in  the  processes  for  assessing  new-drug  technology  and  new 
indications  for  existing  drugs,  certain  assessment  components  are  predictable.  Steiner  et  al.  (1996) 
found  that  many  plans  reviewed  medical  journals,  FDA  reports,  and  professional  association 
information,  and  sought  the  opinions  of  local  experts.  Yet,  we  found  that  each  plan  undertook  this 
activity  independently,  resulting  in  considerable  redundancy  and  inefficiency.  Matuszewski  (1997) 
described  the  basic  components  of  technology  assessment  in  this  setting,  including  who  does  it,  how 
it  is  done,  and  its  common  problems.  He  discovered,  as  we  did,  that  plans  often  find  that  there  is 
insufficient  evidence  on  which  to  base  a  decision,  particularly  early  in  the  technology’s  life  cycle. 

Early  completion  is  particularly  difficult  for  what  Fuchs  and  Garber  (1990)  call  the  “new  style  of 
technology  assessment,”  which  goes  beyond  safety  and  efficacy  to  analyze  comparative  cost- 
effectiveness  and  other  effects,  such  as  an  impact  on  the  labor  force.  Fuchs  and  Garber  also 
describe  concerns  about  the  trustworthiness  of  published  assessments  of  new  technologies  and  the 
incentives  and  rigor  of  the  investigators.  Many  of  our  respondents  echoed  these  concerns.  They 
wondered  about  the  dependability  of  assessments  sponsored  by  manufacturers  (i.e.,  questions  of 
internal  validity)  and  the  possibility  that  clinical-trial  experience  may  not  reflect  the  drug’s  use  in 
“the  real  world”  (i.e.,  questions  of  the  external  validity^®),  and  cited  these  concerns  as  reasons  they 
prefer  to  see  a  period  of  market  experience  before  completing  a  formal  evaluation  of  a  drug. 

In  general,  our  interviewees  indicated  that  their  health  plans  and  provider  organizations 
commonly  encounter  the  same  problems  with  technology  assessment  that  Matuszewski  (1997) 
described:  lack  of  evidence,  changing  standards  of  evidence,  inconsistent  evidence,  lack  of  agreement 
on  how  to  perform  an  assessment,  special  interests,  and  variation  in  the  breadth  and  depth  of 
topics,  new  information,  and  legal  issues. 

Our  interviews  also  identified  differences  in  context  as  a  source  of  variation  between  and 
within  organizations.  The  technology  assessment  hterature  generally  regards  these  variations  in 
the  assessment  process  as  being  problematic.  However,  our  interviewees  were  generally  comfortable 
with  variations  that  they  believed  to  be  appropriate  for  the  differences  in  clinical  and  organizational 
context.  When  asked  how  rigorous  a  new  drug  assessment  needs  to  be,  one  medical  director 
summed  up  the  other  interviewees’  viewpoint  succinctly:  “we  do  enough  assessment  to  make  a 
decision.”  He  and  others  expressed  concern  that  a  single  academic  standard  for  assessing  new 
drugs  or  indications  would  be  too  time-consuming,  resource-intensive,  and  slow  to  be  useful  in  the 

10  This  is  really  an  issue  of  whether  results  obtained  from  selected  populations  treated  in  specialized 
settings  can  be  generalized  to  the  typical  patients  in  usual  settings. 
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“real  world.”  However,  all  those  we  interviewed  thought  a  great  deal  could  be  done  to  make  the 
assessment  process  more  consistent  and  efficient.  Many  expressed  a  need  for  better  and  timelier 
information,  and  for  help  in  increasing  the  quality  of  decisionmaking  without  making  the  process  too 
slow  or  costly. 

Organizations  responsible  for  pharmacy  benefits  sometimes  choose  to  evaluate  several 
different  aspects  of  a  new  drug,  such  as  its  clinical  effectiveness,  safety,  cost  of  treatment,  cost- 
effectiveness,  or  influence  on  quality  of  life.  Chnical  effectiveness  and  safety  assessments  usually 
begin  with  a  literatxire  review  and  synthesis  of  the  available  information  on  the  drug  in  question. 
This  process  may  either  be  performed  by  members  of  the  organization  or  purchased  from  an  external 
source.  When  conducting  an  in-house  evaluation,  the  health  plans  we  interviewed  either  assigned  a 
staff  member  (often  a  chnical  pharmacist)  to  perform  the  literature  reviews  or  a  subcommittee  of 
their  P  &  T  committee  with  expertise  relevant  to  the  condition  that  the  new  drug  is  intended  to 
treat.  Otherwise,  they  purchased  reviews  from  PBMs  and  other  organizations. 

Literature  reviews  utihze  a  range  of  sources,  including  peer-reviewed  journal  articles, 
unpubhshed  articles,  government  documents,  information  from  professional  organizations,  drug 
company  brochures,  and  reports  from  assessing  organizations  (Steiner  et  al.,  1996;  Luce  and  Brown, 
1995;  Lyles,  Luce,  and  Rentz,  1997;  interviews).  However,  the  way  in  which  this  information  is 
used  and  its  perceived  relevance  vary.  In  a  1996  survey  of  physicians,  Steiner  et  al.  found  that 
decisionmakers  at  large  health  plans  were  twice  as  likely  as  those  from  small  health  plans  to  rely 
on  peer-reviewed  medical  journals  in  developing  evaluations.  The  survey  also  found  that  such 
decisionmakers  relied  most  frequently  on  medical  journals,  followed  by  local  experts,  FDA-approval 
documentation,  and  information  from  plan  associations.  A  subsequent  survey  of  decisionmakers  at 
51  health  plans  confirmed  this  general  ranking  of  sources:  The  plans  reHed  most  frequently  on  peer- 
reviewed  literature,  followed  by  government  reports,  then  industry  evaluations  (Lyles,  Luce,  and 
Rentz  1997).  Reports  from  other  HMOs,  PBM  pubfications,  and  non-peer-reviewed  literature  were 
less  popular.  However,  the  survey  respondents  ranked  PBM  assessments  first  in  importance  to  the 
decision  made,  followed  in  order  by  assessments  from  other  HMOs,  peer-reviewed  hterature, 
industry  evaluations,  non-peer-reviewed  literature,  and  government  reports  (Lyles,  Luce,  and 
Rentz,  1997).  This  ranking  suggests  that  comprehensiveness  and  timeliness  are  major 
determinants  of  importance.  Other  surveys  have  also  found  that  the  timehness  of  the  information 
can  drive  the  relative  importance  of  the  source,  possibly  owing  to  the  need  to  make  coverage 
decisions  prior  to  the  publication  of  peer-reviewed  articles  (Steiner  et  al.,  1996;  Luce  and  Brown, 
1995). 
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The  medical  directors  we  interviewed  expressed  an  interest  in  meta-analysis/^  but  they 
beheved  it  usually  to  be  impractical  for  evaluating  new  drugs  because  of  the  general  lack  of  relevant 
studies.  These  interviewees  also  expressed  a  general  frustration  over  the  lack  of  clinical-trial  data 
relevant  to  their  pharmaceutical  assessment  needs.  Consistent  with  the  literature,  the  medical 
directors  we  interviewed  preferred  to  make  coverage  decisions  based  on  clinical-trial  data  while 
being  mindful  that  such  data  may  provide  hmited  information  about  the  performance  of  the  drug  in 
real-world  practice  (Steiner  et  al.,  1996;  Towery  and  Perry,  1981). 

Although  health  plans  are  concerned  about  costs  and  they  may  perform  some  economic 
analyses  of  the  impact  of  new  pharmaceuticals,  they  do  not  generally  perform  formal  mathematical 
modeling,  such  as  cost-outcome  or  decision  analyses.  Interviewees  at  only  two  national  health  plans 
indicated  that  they  use  formal  economic  models,  such  as  pharmacoeconomic  studies,  Monte  Carlo 
simulations,  and  projections.  The  other  organizations  we  interviewed  did  not  use  formal  modeling. 
Smaller  organizations  reported  performing  hmited,  “relatively  crude”  or  “back  of  the  envelope” 
calculations  only  for  new  drugs  expected  to  have  a  large  financial  impact.  Cost  modeling  among 
larger  organizations  generally  consisted  of  estimating  the  impact  of  a  new  drug  based  on  the  cost  of 
the  pharmaceutical  and  its  administration  alone.  These  estimates  are  typically  projected  from  the 
“incidence  of  indications”  (the  prevalence  of  the  condition  that  the  drug  is  Ucensed  to  treat);  the 
“expected  market  share”  of  the  drug;  and  the  drug  cost  per  course  of  therapy.  All  of  these  are 
subject  to  variation  in  practice;  for  example,  cost  per  course  is  affected  by  negotiation  and  volume 
discounting. 

The  published  hterature  echoes  these  findings.  For  example,  in  a  survey  of  231 
decisionmakers  at  HMOs  and  indemnity  plans,  Steiner  et  al.  (1996)  found  that  lack  of  data  on  cost- 
effectiveness  constituted  a  major  barrier  to  making  optimal  coverage  decisions.  Luce  and  Brown 
(1995)  also  found  that  HMOs  and  third-party  payers  do  not  perform  cost-effectiveness  analyses,  nor 
do  they  generally  include  such  information  from  other  sources  in  their  coverage  decisions.  They 
noted  that  hospitals  do  perform  some  in-house  financial  analysis;  however,  Luce  and  Brown  did  not 
report  on  the  rigor  of  those  assessments.  In  contrast,  Schulman  et  al.  (1996)  found  that  PBMs 
occasionally  incorporate  findings  on  the  costs  and  outcomes  associated  with  pharmaceutical  use. 
However,  the  scope  of  these  assessments  is  limited  to  pharmaceutical  use  and  does  not  include  the 
larger  budgetary  issues  of  concern  to  the  overall  health  plan. 

Most  organizations  forecast  the  potential  demand  for  new  drugs,  based  solely  on  clinical  and 
epidemiological  data.  These  organizations  do  httle,  if  any,  formal  demand  modeling  to  take  into 


Meta-analysis  aggregates  the  results  of  multiple  studies,  thereby  increasing  sample  size  and  allowing 
more  statistically  powerful  conclusions  regarding  a  common  treatment  or  intervention  effect  across 
studies. 
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account  non-chnical  factors,  such  as  the  attitudes  and  preferences  of  patients  and  physicians.  If 
demand  modeling  is  done  at  all,  it  uses  casual  (“back  of  the  envelope”)  calculations  that  make 
simple  assumptions,  such  as  that  the  demand  for  a  new  drug  will  be  similar  to  that  of  others  in  its 
class  or  that  new  users  will  be  drawn  solely  from  the  currently  diagnosed  population.  These  models 
usually  neglect  the  effect  of  the  new  drug  on  the  size  of  the  diagnosed  population  (i.e.,  by  recruiting 
patients  into  care  or  increasing  provider  propensity  to  make  the  diagnosis),  because  most 
organizations  feel  that  they  have  no  adequate  method  of  determining  this  variable  (Reissman, 
personal  commxuiication,  1999).  Moreover,  the  focus  of  this  type  of  modehng  is  on  the  impact  of  the 
new  drug  on  pharmaceutical  expenditures;  it  does  not  usually  consider  the  overall  impact  of  drug 
use  on  total  medical  expenditures  (Reissman,  personal  communication  1999;  Schweitzer,  1997). 

Medical  directors  generally  recognize  the  need  to  better  understand  demand.  In  more  than 
one  case,  the  individuals  we  interviewed  pointed  to  drug  manufacturers*  increasing  use  of  direct-to- 
consumer  advertising  as  a  significant  cause  of  increased  demand,  and  they  mentioned  Claritin  and 
Viagra  as  specific  examples  of  this  phenomenon.  Several  medical  directors  noted  that  their 
organizations  had  failed  to  anticipate  the  demand  for  new  drugs  such  as  Viagra.  They  also 
complained  that  existing  risk  contracts  do  not  take  the  possibihty  of  new  blockbuster  drugs  into 
consideration,  leading  them  to  fear  the  budgetary  impact  of  drugs  with  wide  appeal  to  patients, 
particularly  those  for  hfestyle  conditions  lacking  a  clearly  defined  medical  diagnosis.  Several 
medical  directors  also  noted  that  PBMs  could  play  a  greater  role  in  demand  modeling  and 
forecasting  because  they  have  access  to  large  amounts  of  claims  data  from  which  trends  could  be 
identified  and  forecasted. 

Our  interviews  revealed  that  the  level  of  rigor  with  which  organizations  gather  information 
from  these  sources  varies.  Oftentimes,  the  formality  of  the  literature  search  can  depend  on  the  level 
of  controversy  or  demand  for  a  drug.  For  instance,  one  medical  director  mentioned  using  a  more 
formal  technique  for  evaluating  Viagra  than  for  other  drugs  because  it  was  considered  a  “lifestyle 
drug’*  and  a  potential  blockbuster.  However,  even  the  most  formal  reviews  performed  by  PBMs 
appear  to  be  much  less  scientifically  rigorous  than  a  full  systematic  review  or  meta-analysis. 

How  Are  Decisions  Implemented  and  the  Results  Monitored? 

All  of  the  organizations  we  interviewed  had  a  defined  process  of  varying  formahty  for 
documenting  and  disseminating  their  decisions  on  new  pharmaceutical  technologies.  If  the  decision 
is  for  controlled  use,  implementing  management-control  tactics  such  as  developing  educational 
programs  requires  a  major  commitment  of  time  and  resources.  Some  of  the  control  tactics  may 
become  steps  in  the  implementation  process  (e.g.,  developing  cHnical  use  guidehnes,  processes,  and 
educational  programs);  others  may  define  the  implementation  plan  (e.g.,  prior  authorization 
required  for  certain  drugs). 
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The  P  &  T  committees  in  the  organizations  we  interviewed  usually  included  both  clinical 
and  administrative  staff,  and  major  recommendations  typically  required  management  approval. 
Two  of  the  national  plans  we  surveyed  had  a  separate,  formal  committee  to  assess  and  plan  the 
implementation  of  coverage  decisions  regarding  new  drugs,  indications,  or  control  policies;  others 
depended  upon  the  normal  management  structure  to  implement  new  policies  and  programs. 

Every  organizational  decision  has  both  intended  and  unintended  consequences.  Therefore, 
the  monitoring  of  results  is  important.  A  management  process  as  important  to  a  managed  care 
organization  as  implementing  changes  in  drug  benefits  or  control  strategies  should  be  subject  to 
systematic  monitoring  of  the  results.  For  this  reason,  it  was  not  surprising  to  find  that  all  of  the 
organizations  we  interviewed  monitored  the  behavior  of  physicians  and  patient  drug  utilization 
before  and  after  a  major  change  in  policy.  Specifically,  for  shaping  behavior,  all  performed  some 
monitoring  of  physicians  compliance  with  formvilary  and  prescribing  poUcy.  Similarly,  all  the  PBMs 
surveyed  by  Schulman  et  al.  (1996)  profiled  physicians  for  comphance  with  the  formulary  and  all 
offered  programs  to  educate  prescribing  physicians  regarding  formulary  issues,  including  face-to-face 
visits  if  necessary.  Luce  and  Brown  (1995)  found  “evidence  that  the  assessments  and  monitoring  of 
physicians*  procedures  are  modifying  physicians*  practice  patterns.** 

Our  interviews  suggest  that  most  organizations  have  a  limited  ability  to  closely  monitor  the 
clinical  or  financial  effects  of  specific  pharmaceutical  decisions.  This  is  also  consistent  with 
Schulman  et  al.  (1996),  who,  after  finding  that  none  of  the  PBMs  they  studied  tracked  the  health 
outcomes  of  patients  and  that  less  than  half  profiled  physicians  on  cost,  concluded  that  “PBMs  are 
unable  to  track  long-term  outcomes  and  costs  of  disease.  .  .  and  .  .  .  until  comprehensive  integrated 
analyses  are  possible,  these  organizations  wiU  lack  the  ability  to  definitively  show  the  effects  of 
their  programs.**  In  that  study,  the  employers  who  engaged  PBMs  to  manage  pharmacy  benefits 
were  unaware  of  formulary-selection  criteria  and  generally  had  no  way  of  monitoring  the  effects  of 
the  PBM  on  employees*  overall  health  or  total  costs.  Although  health  plans  and  medical  groups 
generally  reported  that  they  monitored  the  type  and  costs  of  drugs  being  prescribed,  none  affirmed 
having  routine  processes  to  assess  the  overall  clinical  and  financial  impact  of  pharmaceutical 
technology  decisions. 

Summaiy 

Variation  in  locus,  timing,  rigor,  resource  intensity,  and  type  of  pharmaceutical  technology 
assessment  is  the  rule  for  managed  care  organizations.  Much  of  this  variation  is  tied  to  the  type  of 
organization  and  its  perceived  needs.  In  turn,  organizational  type  and  perceived  needs  are  closely 
tied  to  the  organization*s  available  range  of  contractually  determined  decision  and  control  options, 
and  the  level  of  controversy  around  and  demand  for  the  drug.  Current  approaches  can  result  in  an 

adequate  response  for  the  decision  at  hand,  but  the  overall  effort  is  often  inappropriately  small  to 
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achieve  optimal  clinical  or  financial  results,  generally  because  of  hmitations  on  resoxirces  or  skill  mix 
at  the  managed  care  organization  or  because  the  original  questions  are  drawn  in  too-narrow  terms. 

The  situation  can  be  improved  with  standardized  methods  and  better  information.  The 
organizations  we  spoke  with  all  indicated  a  desire  for  both.  The  next  section  presents  a 
standardized  framework  and  describes  methods  that  can  be  used  successfully  within  that 
framework. 
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3.  A  Framework  and  Approaches  for  Future  Practice 

Introduction 

As  detailed  above,  our  review  of  the  literat\ire  and  interviews  revealed  that  managed  care 
organizations  generally  lack  the  institutional  structures  and  processes  to  accurately  assess  the 
probable  overall  clinical  and  financial  impact  of  new  pharmaceuticals.  With  several  notable 
exceptions,  P&T  committees  perform  pharmaceutical  evaluations.  These  committees  are  focused  on 
an  essential  but  hmited  set  of  issues  such  as  safety,  efficacy,  and  (usually  acquisition)  cost.  They 
are  not  charged  with  broader  issues,  such  as  the  effects  of  new  pharmaceutical  technologies  on 
overall  care  processes,  patient’s  health  outcomes,  and  the  organization  s  financial  viabihty ,  and 
would  be  ill-prepared  to  consider  them.  Moreover,  they  are  sometimes  remote  from  decisionmaking. 

Many  of  the  organizations  that  we  interviewed  agreed  that  the  evaluation  process  could  be 
improved.  To  achieve  this  improvement,  most  beheved  that  they  needed  the  following. 

•  A  structured  process  for  assessing  pharmaceutical  technology  that  would  improve  the  sharing  of 
information  within  and  across  systems,  provide  a  rational  basis  for  decisionmaking,  and 
increase  transferabihty  of  best  practices. 

•  Unbiased  resource(s)  to  help  identify,  locate,  and  evaluate  the  strength  of  evidence  supporting  a 
pharmaceutical  technology  relative  to  its  alternatives,  as  well  as  information  on  special  features 
that  would  affect  compliance,  associated  clinical  or  administrative  costs,  and  quality  of  life. 

•  Reference  materials  for  evaluating  pharmaceutical  technology,  coverage  options,  and 
decisionmaking  processes,  including  reports  from  professional  and  industry  groups,  government 
agencies,  proprietary  technology  assessment  organizations,  and  other  sources. 

•  Timelier,  standardized  information  on  the  chnical  effectiveness,  costs,  and  cost-effectiveness  of 
drugs  in  real-world  settings,  relative  to  alternatives. 


The  prescriptive  portion  of  this  report,  which  is  aimed  at  managed  care  organizations 
interested  in  improving  their  evaluative  efforts,  focuses  on  the  first  point.  It  outhnes  a  structured 
framework  for  assessing  pharmaceutical  technologies.  Appendix  A  and  the  References  section 
approach  the  next  two  points  by  hsting  potentially  useful  reference  materials.  The  last  point,  the 
data  issues  specific  to  pharmaceutical  manufacturers,  is  beyond  the  scope  of  this  report. 

A  Framework  for  Decisionmaking 

The  ideal  process  for  making  a  pharmaceutical  coverage  decision  works  within  data  and 
budgetary  limitations,  and  is  explicit — documented,  prescribed,  standardized;  valid  ^uses 
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legitimate  and  correct  techniques;  and  reliable — produces  approximately  the  same  results  under 
identical  situations.  An  explicit  process  helps  guard  against  biases  that  may  be  hidden  in  ad  hoc 
processes,  and  which  may  vary  from  decision  to  decision.  A  valid  process  will  maximize  the 
likelihood  of  making  the  correct  coverage  decision  (i.e.,  the  decision  most  likely  to  optimize 
outcomes).  A  reliable  process  ensures  that  coverage  decisions  for  similar  drugs  are  similar.  With 
these  goals  in  mind,  we  combined  the  comments  of  our  interviewees,  findings  from  the  literature, 
and  our  own  judgment  to  produce  the  conceptual  framework  depicted  in  Figure  3.1. 

The  steps  should  generally  be  completed  in  the  following  sequence: 


•  Characterize  the  drug  and  the  disease.  Gather  information  on  the  drug  s  relationship  to  the 
marketplace  (e.g.,  therapeutic  alternatives,  clinical  need)  and  on  its  potential  risks  and  benefits, 
especially  in  relation  to  existing  therapies.  Gather  information  on  the  disease’s  present  and 
potential  future  prevalence,  and  its  impact  on  quality  and  duration  of  life,  and  medical  care 
expenditures.  At  this  stage  in  the  process,  the  decisionmaker  needs  to  identify  the  relevant 
issues,  not  to  verify  or  quantify  this  information. 

•  Characterize  the  decisionmaking  organization.  Specify  the  nature  of  the  organization  (e.g., 
health  plan,  provider  organizations),  its  structure  and  contractual  obligations,  and  other  factors 
that  affect  the  extent  to  which  and  the  means  by  which  it  can  promote  or  limit  the  use  of 
pharm  aceuticals . 

•  Define  the  decision  options.  Combine  knowledge  of  the  pharmaceutical  and  the  decisionmaking 
organization  to  identify  the  range  of  available  mechanisms  for  affecting  the  use  of  the  drug  in 
question. 

•  Define  the  analysis  needed  to  make  a  decision.  Define  the  criteria  for  determining  which  drugs 
will  undergo  formal  analysis,  set  the  appropriate  level  of  resources  to  devote  to  the  assessment, 
identify  the  agent  who  will  perform  the  assessment  (i.e.,  the  make  or  buy  decision),  establish 
the  timing  of  the  assessment,  and  specify  the  assessment  methodology.  We  group  these  tasks 
because  they  are  interrelated.  For  instance,  the  degree  of  rigor  applied  to  the  analysis  depends 
upon  the  level  of  resources  devoted  to  it  and  vice  versa. 
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Figure  3.1— The  Decisionmaking  Framework 


•  Perform  the  assessment.  In  many  cases,  this  will  be  the  most  time-consuming  step,  even 
though  the  task  should  be  weU  understood  by  the  time  it  is  reached.  We  classify  the  main 
relevant  methodologies  for  performing  the  assessment  into  three  major  types.  Expert  opinion 
encompasses  both  the  usual  unstructured  consultation  and  more-formahzed  consensus  methods. 
Evidence  assessment  can  also  be  conducted  with  varying  degrees  of  formahty  and  quantitation, 
but  in  all  cases,  it  refers  to  evaluations  that  rely  mainly  on  studies  reported  in  peer-reviewed 
literature,  industry  pubhcations,  or  scientific  meetings.  Evidence  assessment  is  distinct  from 
the  information  gathering  that  takes  place  throughout  the  process  in  that  it  moves  from 
identification  of  issues  to  evaluation  of  evidence.  Mathematical  assessment  includes  evaluations 
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that  attempt  to  quantify  the  impHcations  of  decision  options;  it  relies  on  analysis  of  existing 
data,  with  an  eye  to  predicting  the  effects  of  a  course  of  action  on  an  outcome  of  interest. 

•  Make,  implement,  and  evaluate  the  decision.  Use  the  results  of  the  assessment  to  make  the 
decision  regarding  coverage,  controls,  and  implementation  tactics.  Operationalize  the  coverage 
decision,  including  disseminating  the  decision  and  implementing  the  controls.  Monitor  and 
evaluate  the  results  of  the  decision  by  providing  mechanisms  for  assessing  outcomes  and  for 
periodic  or  continuous  improvement  in  the  coverage  decision.  While  these  later  steps  are  crucial 
to  the  ultimate  value  of  the  evaluation  process,  they  are  highly  individualized  and  lie  beyond  the 
scope  of  this  report. 


Each  step  is  described  more  fully  in  the  following  subsections. 

The  Steps  in  the  Decisionmaking  Process 

Characterize  the  Drug  and  the  Disease 

Gathering  information  to  “define”  or  “profile”  the  drug  under  consideration  provides  the  basis 
for  decisionmaking.  This  step  does  not  explicitly  require  formal  processes  and  systematic  review; 
however,  for  consistency  over  time,  the  process  must  be  constructed  so  that  similar  information  is 
gathered  on  each  product  under  review.  At  a  minimum,  the  information  gathered  should  include 


•  the  potential  benefits  and  risks  of  treatment 

•  the  position  of  the  drug  in  the  current  market  relative  to  alternative  treatments  or  to  no 
treatment 

•  costs  associated  with  its  use 

•  unique  characteristics  of  the  drug  that  would  affect  an  organization’s  decision  about  that  drug. 
Table  3.1  suggests  issues  relevant  to  each  of  these  four  categories. 
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Table  3.1 — Characterizing  the  Drug 

I.  Basic  identification  of  drug 

Purpose:  To  identify  the  drug,  its  common  indications,  and  usage 

Issues:  Generic  name  and  brand  name 

Therapeutic  class 
Indications  for  use 
Mechanism  of  action 

Routes  of  administration  and  dosage  guideiines 
Monitoring  criteria 

II.  Potential  benefits  and  risks 

Purpose:  To  identify  the  clinical  issues  concerning  the  drug  in  relation  to  its  intended 

treatment  population  and  to  other  treatment  options 
Issues:  Effectiveness 

•  Likeiy  impact  on  disease  morbidity  and  mortality 

•  Cure  or  prevention  rates 

•  Prevalence  of  contraindications 

•  Improvement  in  key  disease  markers 

•  Improvement  in  patient  quality  of  life,  physical  and  social  functioning, 
and  satisfaction  with  care 

Untoward  effects 

•  Incidence  of  serious  events 

•  Reported  and  potential  adverse  drug-drug  interactions 

•  Deterioration  in  patient  quality  of  life,  physical  and  social  functioning, 
and  satisfaction  with  care 

Other 

•  Impact  on  care  process  (e.g.,  effects  on  outpatient  vs.  inpatient 
treatment:  increased  requirements  for  laboratory  monitoring) 

•  Impact  on  total  health  care  expenses  (positive  or  negative) 

•  Impact  on  labor  outcomes  (i.e.,  patient  productivity) 

III  Drug  position  in  current  market 

Purpose:  To  understand  the  larger  treatment  context  for  a  specific  condition  or 

disease 

Issues:  •  FDA  approval  (existing/pending) 

•  New  or  existing  therapeutic  class 

•  Competing  products  or  services 

•  Cost  per  course  of  treatment  (absoiute/relative  to  other  therapies) 
and  per  patient  per  unit  time 

•  Severity  of  condition  treated  (life-threatening/chronic/benign) 

•  Patient-initiated  demand 

•  Likely  product  life  cycle  (acceptance/plateau/decline) 
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Table  3.1 — continued 

IV.  Additional  costs 

Purpose:  To  understand  the  total  direct  costs  of  drug  administration 

Issues:  Special  ordering 

•  Product  availability  (readily/lottery /other  mechanism) 

•  Distribution  source  (manufacturer/wholesaler/single  source) 

Handling  (shipping/storage/reconstitution) 

Administration 

•  Packaging  (consistent  with  common  prescription  sizes/benefit 
coverage  guidelines) 

•  Routes  and  circumstances  of  administration 

•  Monitoring  (laboratory/diagnostic  testing  for  efficacy  or  adverse 
events) 

Manufacturer's  policies  (e.g.,  discounts,  rebates,  returned  goods) 

V.  Unique  characteristics 

Purpose:  To  determine  whether  there  is  something  unique  about  the  drug,  or  the 

organization’s  relationship  to  it,  that  should  affect  a  decision 

Issues:  Plan  benefit  language 

•  Little  evaluation  may  be  needed  for  clearly  excluded  drugs 
Uniqueness/dominance  of  drug  in  therapeutic  class 

•  Relevance  of  cost-effectiveness  of  dominant  drug  for  serious  disease 
Nature  of  demand  prior  to  and  after  introduction 

•  From  important  clients/regulators 

•  From  providers 

•  From  patients 

Likelihood  of  becoming  a  "blockbuster,”  or  precedent-setting,  drug 

•  Unusually  expensive 

•  Major  impact  on  cost,  quality 

•  Precedent-setting  for  future  coverage  or  practice 


In  contrast,  characterizing  the  disease  is,  at  least  initially,  a  more  straightforward, 
conceptually  simple  task  of  hsting  epidemiological,  cHnical,  and  economic  parameters.  Relevant 
epidemiological  parameters  include  the  present  and  projected  future  incidence,  dimation,  and 
prevalence.  Clinical  parameters  include  the  severity  and  time  course  of  direct  and  secondary 
morbidity  and  mortality,  and  impacts  on  functional  and  health  status.  Economic  parameters 
include  the  type  and  intensity  of  related  use  of  health  care. 
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Characterize  the  Decisionmaking  Organization 

The  organizational  context — structure,  philosophy,  financial  incentives,  market  position, 
existing  relationships,  and  the  population  served — affects  decisions  about  new  pharmaceutical 
technology.  Understanding  the  interaction  between  an  organization  and  the  drug  under 
consideration  provides  a  valuable  basis  for  defining  the  assessment  and  decisionmaking  process. 

For  example,  if  important  chents,  such  as  major  employers,  have  demanded  coverage  for  a  specific 
drug  that  appears  to  be  excluded  by  the  benefit-plan  language,  the  organization  may  decide  that  a 
full  assessment  rather  than  a  simple  reiteration  of  existing  policy  is  necessary.  If  the  drug  creates  a 
new  class  of  coverage  and  is  extremely  expensive,  the  organization  may  decide  to  include  extensive 
financial  modehng  in  the  assessment,  as  weU  as  to  involve  other  parts  of  the  organization  in  the 
decisionmaking  process.  Many  of  these  organizational  issues  need  to  be  addressed  only  periodically; 
the  relevant  information  can  be  applied  to  multiple  drug  assessments  over  time.  Table  3.2  outlines 
various  issues  relevant  to  this  step. 

Define  Decision  Options 

Physicians  and  their  patients,  not  managed  care  organizations,  have  direct  control  over  the 
prescribing  and  use  of  pharmaceuticals.  However,  managed  care  organizations  influence  the 
behavior  of  physicians  and  patients  through  rules,  enforcement  mechanisms,  and  financial 
incentives — mechanisms  that  define  the  range  of  decision  options  available  between  the  two 
extreme  options  of  (a)  drug  exclusion  and  (b)  promotion  of  unrestricted  use.  The  ability  of  the  service 
delivery  organization  to  implement  these  mechanisms  depends  upon  its  level  of  integration,^^  the 
sophistication  of  its  information-management  systems,  and  the  competitiveness  of  its  local  market. 
Some  of  the  more  common  control  mechanisms  are  as  follows: 


•  Formularies.  Traditionally  used  to  refer  to  a  list  of  covered  or  available  drugs,  formulary  now 
usually  refers  to  a  program  that  includes  the  fist  of  and  procedures  for  obtaining  covered  and 
non-covered  drugs  under  varying  circumstances.  Formulary  programs  create  these  Hsts  and 
procedures  according  to  the  effectiveness,  safety,  and  cost  of  the  covered  drugs.  Formulary 
programs  vary  in  the  number  of  covered  drugs,  the  restrictions  placed  on  covered  drugs,  and  the 
ease  with  which  physicians  can  obtain  coverage  for  off-formulary  drugs. 


^  Level  of  integration  reflects  how  many  functions  are  incorporated  into  the  organization’s  structure:  At  a 
low  level,  local  doctors  refer  to  other  practices  and  pharmacies;  at  a  high  level,  a  large  organization  such 
as  Kaiser  owns  facilities  and  pharmacies  and  refers  only  within  the  organization. 
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Table  3.2 — Characterizing  the  Decisionmaking  Organization 

I.  Organizational  structure 

Purpose:  To  delineate  important  aspects  of  the  organization  that  could  affect  drug 
coverage 

Issues:  Type  of  organization  (e.g.,  health  plan,  pharmacy  benefits  management 
company,  medical  group,  hospital) 

Decisionmaking  authority  for  drug  coverage  (yes/no/shared) 

•  Locus  of  responsibility  (e.g.,  medical  director/P  &  T  committee) 

•  Existing  evaluation  procedures 

•  Ability  to  influence  decisions  in  other  organizations  that  affect  own 
organization 

II.  Institutional  characteristics 

Purpose:  To  identify  important  institutional  characteristics  that  could  affect 
pharmaceutical  evaluation 

issues:  Organizational  philosophy  regarding  drug  coverage/policy/controls 
Relative  importance  of  drug  coverage/policy/costs/controls 

Involvement  in  managing  physician  prescribing  behavior  (e.g.,  prospective 
review  and/or  authorization  of  specified  drugs) 

Formulary  (yes/no;  type:  open/closed/tiered) 

Cost-sharing  features  of  prescription  drug  plans 
Percent  of  premium  spent  on  drugs 

•  Absolute  amount 

•  Direction  of  change  (increasing/decreasing/steady) 

•  Management  goals 

III.  Financial  incentives 

Purpose:  To  understand  the  financial  constraints  and  incentives  under  which  the 
organization  operates  with  regard  to  drug  coverage 

Issues:  Degree  to  which  the  organization  is  at  risk  for  drug  costs  (fully/partially) 

Scope  of  financial  responsibility  (e.g.,  prescription  drugs,  office- 
administered  drugs,  administrative  costs) 

Extent  to  which  other  contracting  organizations  assume  financial 
responsibility  for  drug  costs 

Degree  to  which  existing  financial  arrangements  are  relevant  to  drug 
assessment,  policy,  timing,  priorities,  and  control  options 
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Table  3.2— continued 

IV.  Local  health  care  market 

Purpose:  To  understand  the  competitive  factors  at  work  in  the  market(s)  in  which  the 
organization  operates 

Issues:  Overall  degree  of  market  competition 

Relative  importance  of  drug  benefits/range  of  drug  choices  in  marketplace 
Relative  importance  of  generosity  vs.  cost  of  benefits  in  the  marketplace 

Competitors’  relevant  drug  policy  strategies  (e.g.,  early  adoption  of  new 
drug  technologies) 

Total  premium  and  prescription  drug  costs  relative  to  those  of  competitors 

V.  Current  relationships  with  manufacturers 

Purpose:  To  delineate  how  existing  relationships  with  drug  manufactures/vendors  will 
influence  organizational  policies 

Issues:  Type  of  relationships  with  drug  manufacturers  and  distributors  (e.g., 
exclusive  or  preferential) 

Contractual  obligations 

Degree  of  dependence  on  others  (e.g.,  drug  manufacturers,  PBMs,  medical 
groups,  health  plans)  to  finalize  or  implement  drug  policy  decisions 

VI.  Population  served 

Purpose:  To  describe  the  demographics  of  the  covered  population  and  identify  any 
special  needs 

Issues:  Overall  population  demographics  (age/sex  distribution) 

Type  of  insurance  coverage  (e.g.,  employer-sponsored.  Medicare, 

Medicaid) 

Special  subpopulations  (e.g.,  seniors,  medically  indigent,  chronically  ill) 

•  Drug  policy  issues  specific  to  these  populations 
_ •  Regulatory  requirements  specific  to  these  populations _ 

•  Restrictions  on  covered  drugs  take  several  forms.  Dose  limits  restrict  the  amount  of  the  drug 
covered  (e.g.,  limiting  coverage  of  Viagra  to  a  specified  number  of  pills  per  month).  Step-therapy 
programs  ensure  that  certain  therapeutic  substitutes  are  tried  before  a  more  expensive  and/or 
toxic  drug  is  prescribed.  Limiting  medical  indications  restricts  overall  use  by  allowing  specific 
drugs  to  be  used  for  only  particular  medical  indications  (e.g.,  permitting  use  of  Retin  A  for  acne 
but  not  for  wrinkles). 

•  Guidelines  outhne  circumstances  under  which  a  drug  is  recommended  for  use.  To  increase 
physician  comphance,  guidehnes  are  often  developed  internally  with  provider  input,  but 
successful  implementation  still  requires  substantial  resources. 

•  Prior  authorization  requires  physicians  to  obtain  permission  to  prescribe  a  particular  drug 
before  prescribing  it.  This  requirement  helps  to  ensure  compliance  with  guidehnes,  usage 
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restrictions,  and  step-therapy  programs.  Utilization  review  has  similar  objectives,  but  it  is  used 
retrospectively  to  determine  whether  physicians  comphed  with  guidelines. 

•  Differential  co-pavments  create  financial  incentives  for  patients  to  choose  less-expensive  drugs 
over  chnical  substitutes.  For  example,  generic  drugs  may  require  lower  co-payments  than  do 
brand-name  drugs. 

•  Promulgation  and  promotion  of  guidelines  or  care  oaths  encouraging  use  of  the  new  drug. 

Define  the  Analysis  Needed  to  Make  the  Decision 

This  step  encompasses  several  elements,  including  defining  the  criteria  for  selecting  which 
drugs  will  undergo  formal  analysis  and  deciding  upon  the  timing,  type,  resovurce  intensiveness,  and 
appropriate  methodology  for  the  analysis. 

The  literatxire  suggests  that  formal  analyses  be  reserved  for  important  drugs,  which  can  be 
defined  as  those  that  will  be  used  by  large  numbers  of  patients  or  have  a  high  cost  per  coiorse  of 
treatment,  a  high  likelihood  of  adverse  events,  controversial  uses,  or  large  variations  in  use  across 
providers  (Matuszewski,  1997).  Eddy  (1989)  proposes  similar  criteria  but  adds  consideration  of  the 
expected  outcome  of  the  assessment,  the  potential  to  influence  the  use  of  the  technology,  and  the 
associated  changes  in  individual  and  population  outcomes.  Similarly,  Donaldson  and  Sox  (1992) 
proposed  a  quantitative  firamework  for  priority  setting  that  considered  prevalence,  morbidity 
(disease  burden),  cost,  variation  in  use,  and  the  potential  to  change  outcomes  and  costs  and  to 
inform  social  and  ethical  issues.  Phelps  and  Mushhn  (1988),  who  were  considering  pharmaceutical 
technology,  have  suggested  using  a  “back-of-the-envelope  ”  feasibility  calculation  to  determine 
whether  the  decision  is  sensitive  to  the  potential  results  of  a  full  assessment. 

Extending  this  approach  to  pharmaceuticals,  one  might  do  similar  calculations  to  determine 
whether  a  new  and  highly  effective  drug  might  be  more  cost-effective  than  less  cHnically  effective 
existing  agents  under  a  plausible  range  of  assumptions.  If  so,  the  drug  would  be  better  and 
cheaper,  and  a  decision  to  adopt  it  can  usually  be  made  without  further  investigation. 

Governmental  and  professional  organizations  use  criteria  for  funding  technology 
assessments  that  could  be  adapted  for  use  for  setting  assessment  priorities.  The  U.K.  Medical 
Research  Council  considers  the  importance  of  the  disease,  proposed  methodology  of  assessment, 
timehness,  generalizabihty,  compatibility,  and  value  for  money  (Harper,  Townsend,  and  Buxton, 
1998).  The  National  Institutes  of  Health  Office  of  Medical  Apphcations  of  Research  (OMAR)  uses 
medical  importance,  scientific  controversy  amenable  to  consensus  research,  availabihty  of  data, 
abihty  to  be  objectively  evaluated,  and  appropriateness  of  timing  of  the  assessment  (Eddy,  1989). 
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Issues  of  the  staffing,  timing,  and  the  methodological  rigor  of  a  drug  assessment  should  vary 
not  only  by  the  nature  of  the  outcomes  of  most  interest  but  also  by  organizational  situation — its 
legal/regulatory  requirements,  existing  benefit  structures,  coverage  preferences  among  similar  drugs, 
competitive  environment  (i.e.,  the  influence  of  its  decisions  on  competitor  or  client  behavior),  the  a 
priori  range  of  possible  clinical  and  financial  outcomes  from  adoption,  and  availability  of  resources. 
Therefore,  we  touch  only  briefly  on  these  topics.  Other  considerations  include  the  similarity  of  the 
assessment  in  question  to  available  previous  assessments,  and  the  availabihty  of  in-house  staff  or 
outside  consultants  to  perform  the  assessments.  Timing  issues  include  determining  when  in 
chronological  time  or  in  the  life  cycle  of  the  drug  to  perform  an  assessment  or  re-assessment,  and  the 
amount  of  time  to  devote  to  it. 

Perhaps  the  most  important  of  these  decisions  is  the  choice  of  methodology  for  the  final 
evaluation.  Which  evaluation  methods  are  appropriate  to  technology  assessment  vary  with  the 
researcher  (Garber,  1994;  Matuszewski,  1997;  Goodman,  1998).  We  find  it  useful  to  classify  the 
most  relevant  methods  into  three  groups:  evidence  assessment,  expert  opinion,  and  mathematical 
assessment. 

Evidence  Assessment.  Until  recently,  evidence  assessment  meant  that  a  scientist  who 
collected  evidence  in  what  might  be  characterized  as  a  nonsystematic  manner — ^such  as  searching 
only  one  hbrary  database  and  not  supplementing  that  search  by  contacting  experts  (see  the 
“Perform  the  Assessment”  subsection  below  for  ways  to  do  a  search  systematically) — ^woiild  then 
summarize  the  results  in  narrative  form.  This  type  of  informal  review  usually  wiU  take  place  in  the 
early  stages  of  the  decisionmaking  process.  Although  useful,  such  reviews  are  idiosyncratic,  mix 
opinion  with  evidence,  and  both  collect  and  summarize  evidence  in  a  non-reproducible  way.  The 
evidence-based  medicine  movement,  which  emphasizes  data-driven  care,  has  developed  better 
means  of  evaluating  evidence.  Managed  care  decisionmakers  are  likely  to  concentrate  on  the  usual 
chnical  evidence  for  assessing  pharmaceuticals:  completed  scientific  studies. This  evidence  is 
best  obtained  and  summarized  by  a  full  evidence  assessment,  here  defined  as  a  systematic  review. 

To  quahfy  as  a  systematic  review,  an  evidence  assessment  must  comprehensively  collect,  sift, 
classify,  and  summarize  the  existing  knowledge  either  qualitatively  or  quantitatively,  using  meta- 
analytic  techniques  (Sackett  et  al.,  1997).^^  If,  as  is  often  the  case,  available  studies  do  not  provide 


13  NOTE:  evidence  assessment  can  include,  for  example,  the  secondary  analysis  of  patient  databases. 

14  Some  would  say  that  the  standards  of  the  evidence-based  medicine  movement  require  an  unreasonably 
comprehensive  approach  because  of  the  low  returns  to  a  very  extensive,  compared  to  merely  extensive, 
effort  (e.g.,  looking  at  the  informal  literature  in  non-Western  languages). 
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adequate  information  for  a  targeted  evidence  report,  decisionmakers  should  extend  available 
information  using  expert  or  mathematical  assessments.^^ 

Evidence  assessment  can  be  most  useful  when  addressing  a  focused  question  supported  by 
high-quality  evidence;  however,  the  requirements  for  thoroughness  and  reproducibility  can  be  taxing 
if  performed  to  the  highest  standard.  Fortunately,  rapidly  increasing  numbers  of  useful  systematic 
reviews  can  be  found  through  routine  searches  of  the  medical  literature,  as  well  as  by  contacting 
sponsors  such  as  the  Cochrane  Collaboration,  an  international  nonprofit  organization 
(www.cochrane.org),  and  the  Agency  for  Healthcare  Research  and  Quality  (AHRQ;  formerly  the 
Agency  for  Health  Care  Pohcy  and  Research,  AHCPR),  which  sponsors  12  Evidence-Based  Practice 
Centers  across  the  United  States  (www.ahcpr.gov/clinic/epc). 

Expert  Assessment.  Expert  assessment  has  been  used  to  refer  to  the  common  practice  of 
asking  for  the  opinion  of  an  informed  expert  or  two.  This  simple  approach  should  be  reserved  for 
straightforward  decisions,  such  as  a  medical  director  choosing  to  adopt  a  close  substitute.  It  is  too 
narrowly  based,  idiosyncratic,  and  irreproducible  for  decisions  about  important  new  technologies. 
Rather,  expert  assessment  for  significant  new  pharmaceutical  technologies  is  best  obtained  from  a 
group  process  that  draws  upon  a  wide  range  of  direct  knowledge  and  experience. 

Expert  consensus  methods  are  an  appropriate  means  of  decision  support  when  the  scientific 
literature  contains  few  high-quahty  research  studies  and  a  vahd  research  synthesis  cannot  be 
conducted — a  situation  that  often  occurs  during  the  early  or  “emerging”  phase  of  a  technology  but 
that  also  occurs  when  it  is  necessary  to  generalize  to  specific  patient  groups  and  clinical  situations 
that  are  not  directly  addressed  even  when  the  scientific  literature  is  well  developed.  However,  in  all 
cases,  it  is  important  to  remember  that  consensus  development  is  a  process  for  making  policy 
decisions,  not  a  scientific  method  for  creating  new  knowledge  (Phelps,  1993). 

Mathematical  Models.  Mathematical  models  represent  the  decision  to  be  made  in 
quantitative  terms  and  that  must  predict  the  outcomes  of  alternatives,  often  using  the  results  of 
evidence  and  expert  assessments  as  inputs.  More  generally,  models  are  decisionmaking  aids  that 
represent  processes  in  physical,  graphical,  or  mathematical  terms.  Well-constructed  models  expose 
“structural  relationships  among  key  variables”  (Stokey  and  Zeckhauser,  1978).  Models  are  uniquely 
able  to  reduce  uncertainty  by  using  these  relationships  to  predict  consequences  of  alternative 
strategies  and  are  thus  particularly  useful  when  data  are  incomplete  or  fragmented.  For  example, 
they  can  be  used  for  identifying  the  effects  of  a  drug  on  the  overall  continuum  of  care  when  only 
clinical  data  on  a  subset  of  that  care  (e.g.,  the  acute  effectiveness  of  a  drug  that  shortens  flare-ups  of 


For  instance,  most  studies  will  not  consider  the  particular  population  or  cost  structure  under 
consideration. 
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a  chronic  disease)  are  available.  In  addition,  models  are  useful  for  translating  the  findings  of  other 
assessments  into  results  that  are  meaningful  for  a  specific  organization. 

Mathematical  models  are  particularly  suited  for  describing  the  likely  effect  of  new  drugs 
under  various  control  mechanisms,  in  part  because  the  key  outcomes  of  pharmaceutical  use  (e.g., 
improved  health,  expenditures)  lend  themselves  to  these  approaches  (Austin  and  Boxerman,  1995). 
Three  tj^es  of  mathematical  models  are  particularly  relevant  to  drug  assessment:  cost-outcome 
analysis,  decision  analysis,  and  demand  assessment. 

Cost-Outcome  Analysis,  Only  this  assessment  method  simultaneously  quantifies  both  the 
costs  and  benefits  of  a  decision.  It  is  especially  useful  when  adoption  of  new  pharmaceutical 
technology  is  expected  to  severely  strain  a  tight  budget.  In  this  context,  the  result  of  a  cost-outcome 
analysis  is  usually  expressed  as  the  ratio  of  the  costs  of  the  decision  made  (e.g.,  add  the 
pharmaceutical  to  the  formulary)  to  benefit  of  the  decision  (e.g.,  life-years  saved).  Cost-outcome 
analysis  can  be  divided  into  three  categories. Cost-effectiveness  analysis  (CEA)  estimates  the  cost  of 
each  unit  of  chnical  improvement  on  similar  outcomes  (e.g.,  life-years  saved).  Cost-utility  analysis 
converts  simileu*  and  dissimilar  outcomes  to  the  common  metric  of  utility,  or  total  net  perceived  value 
of  present  and  future  health.  Cost-benefit  analysis  (CBA),  which  converts  all  costs  and  benefits  to  a 
monetary  value,  is  unpopular  in  the  healthcare  setting  because  of  the  technical  and  ethical  barriers 
to  placing  a  monetary  value  on  health  outcomes. 

The  results  of  cost-outcome  analysis  should  not  be  the  only  consideration  when  making 
coverage  decisions,  because  they  could  point  to  pohtically  untenable  coverage  decisions  (Eddy,  1991). 
Additionally,  the  results  of  a  cost-outcome  analysis  always  require  interpretation.  Individual  drugs 
or  alternative  courses  of  action  are  at  least  as  effective  and  no  more  costly  than  alternative 
strategies,  or  are  less  effective  and  more  costly  (O’Brien  et  al.,  1997).  When  the  decision  to  be  made 
is  clear,  a  formal  cost-outcome  analysis  can  add  value  only  by  quantifying  the  likely  impact  of  the 
pohcy  to  be  adopted.  However,  the  optimal  course  of  action  is  less  clear  when  one  of  the  alternatives 
is  more  effective  and  more  costly,  or  less  effective  and  less  costly. 

Finally,  we  note  that  cost-outcome  analyses  can  simplify  to  cost-minimization  analyses 
when  outcomes  are  similar.  When  evaluating  close  substitutes,  an  assumption  of  similar  outcomes 
may  be  reasonable  and  one  might  start  with  a  cost-minimization  analysis.  However,  this  is  usually 
inappropriate  when  considering  a  significant  new  pharmaceutical,  because  only  simultaneous 


16  We  use  the  term  cost-outcome  analysis  in  order  to  avoid  the  confusion  that  can  arise  from  the  common 
practice  of  using  cost-effectiveness  analysis,  not  only  for  what  we  refer  to  as  cost-effectiveness  analysis,  but 
to  subsume  cost-utility  and  cost-benefit  analysis  as  well. 

17  There  is  some  variation  in  the  specific  definitions  of  cost-outcome  analysis  provided  across  different 
sources.  For  this  report,  we  adopt  the  convention  described. 
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consideration  of  cost  and  outcomes  illuminates  the  possibility  that  a  more  expensive  alternative 
represents  a  prudent  investment. 

Decision  Analysis,  In  pharmaceutical  assessment,  the  benefits  of  a  strategy  in  a  specific 
circumstance  must  be  estimated.  Decision  analysis  is  a  mathematical  method  of  estimating  likely 
consequences  of  an  action  when  the  evidence  is  insufficient  to  directly  support  decisionmaking. 
Sometimes  incorporating  data  on  costs  (Haddix  et  al.,  1996),  decision  analyses  generally  evaluate 
how  a  decision  affects  health  outcomes  (Sox,  Blatt,  and  Higgins,  1988;  Richardson  and  Detsky, 

1995a,  b;  Pauker  and  Kassirer,  1987).  They  are  especially  exphcit  because  they  develop  a  decision 
tree  (see  “Decision  Analysis”  subsection),  which  shows  the  pathways  from  each  decision  point  to  all 
possible  outcomes,  the  probabihty  that  each  outcome  could  occur,  and  the  expected  result  of  a 
decision,  which  combines  pathway  and  probability. 

Demand  Assessment.  Whereas  cost-outcome  assessments  and  decision  analyses  are  usually 
performed  to  determine  the  outcome  of  a  decision  in  a  controlled  environment,  such  as  the  impact  of 
beta-blockers  when  used  by  all  eligible  patients,  that  use,  which,  in  turn,  depends  on  providers 
prescribing  it,  must  be  understood  and  reliably  predicted.  Demand  assessment  investigates  the 
number  of  people  likely  to  take  beta-blockers  under  each  coverage  scenario.  This  type  of  analysis  is 
an  important  component  of  cost  analyses,  resource  planning,  and  benefits  measurements.  Given 
the  characteristics  and  actions  of  providers,  patients,  payers,  service  dehvery  organizations,  and  the 
drug  (Schweitzer,  1997),  demand  models  predict  pharmaceutical  use  (or  realized  demand). 
Pharmaceutical  use  is  too  complex  to  model  in  its  entirety,  but  it  can  be  described  by  models  that 
account  for  drug  initiation,  dosage,  and  duration.  The  most  useful  models  focus  on  the  choices  or 
sequence  of  choices  relevant  to  the  managerial  decision  at  hand.  For  example,  a  health  plan  may  be 
particularly  interested  in  modeUng  the  initiation  of  treatment  if  a  new  drug  is  likely  to  recruit 
patients  into  treatment  for  a  specific  condition;  however,  it  may  be  less  concerned  about  modeling 
comphance  if  the  duration  of  use  is  brief,  the  side  effects  are  minimal,  and  the  benefits  are  high. 

Perform  the  Assessment 

A  managed  care  organization  may  choose  to  use  one  or,  more  likely,  a  combination  of  the 
above  analytic  modalities. 

Evidence  Assessment.  If  performing  a  full  evidence  assessment,  or  systematic  review,  the 
MCO  will  engage  in  a  multi-step  process  in  which  all  relevant  clinical  evidence  is  collected  into  a 
systematic  review  that  summarizes  the  knowledge  about  a  clinical  area  and  identifies  gaps  in  the 
hterature.  It  is  completely  adequate  only  when  it  examines  all  of  the  relevant  clinical  evidence 
systematically  and  reproducibly. 
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Those  conducting  definitive  systematic  reviews  should  consult  an  experienced  medical 
hbrarian  who  is  famihar  with  the  techniques  developed  by  the  consensus  of  experts  participating  in 
the  Cochrane  Collaboration  and  others  rather  than  relying  on  a  simple  Medline  search,  which  is 
likely  to  miss  many  relevant  articles,  including  randomized  chnical  trials  {The  Cochrane  Methods 
Handbook,  1998;  Dickersin,  Scherer,  and  Lefebvre,  1994).  A  full  effort  would  ideally  include 
searches  of  other  databases,  the  non-Enghsh  hterature,  and  the  so-caUed  gray  hterature  of 
conference  proceedings,  and  reports  to  and  from  government.  All  of  these  sources  have  been  shown 
to  increase  yields  and  should  be  covered  to  avoid  bias  (Counsell,  1997 ;  Egger  and  Smith,  1998; 

Egger  et  al.,  1997).  Searching  through  important  sources  and  registries  by  hand  for  mis-  or  un¬ 
indexed  material  should  be  considered  as  well. 

An  adequate  search  process  often  results  in  the  identification  of  a  large  number  of  potential 
sources,  which  must  be  reduced  to  the  truly  useful  by  a  reproducible  process  that  clearly  delineates 
inclusion  and  exclusion  criteria.  In  the  Southern  California  Evidence  Based  Practice  Center,  we 
follow  the  common  practice  of  generating  criteria  to  apply  sequentially  to  the  fists  of  titles, 
abstracts,  and  articles  to  determine  which  to  review  at  each  successive  step  in  the  winnowing 
process  (The  Cochrane  Library,  1999).  Of  course,  such  an  extensive  effort  is  not  always  appropriate 
or  possible.  Even  when  the  intent  is  not  to  be  definitive,  a  systematic  and  reproducible  approach 
addressing  the  most  important  databases  and  other  sources  is  most  likely  to  uncover  the  most 
important  information. 

To  assess  the  quality  of  the  evidence,  the  evidence  must  first  be  categorized  according  to  a 
hierarchy  of  quality,  such  as  that  of  the  widely  used  Canadian  Task  Force  system.  This  system 
classifies  study  designs  into  (I)  randomized  controlled  trial;  (II- 1)  nonrandomized  controlled  trial;  (II- 
2)  well-designed  cohort  or  case-control  study;  (II-3)  multiple  time-series  studies  with  or  without  the 
intervention;  and  (IV)  expert  opinion  (Canadian  Task  Force  on  the  Periodic  Health  Examination, 
1979).  Once  classified,  the  quality  of  each  individual  study  needs  to  be  assessed  according  to  the 
strength  of  the  design  and  the  execution  of  that  design.  Weaker  classes  of  studies  and  weaker 
studies  within  classes  may  be  removed  fifom  further  consideration  or  deleted  in  sensitivity  analyses 
to  assess  their  impact.^* 

After  quality  assessment,  studies  need  to  be  organized  into  clinically  sensible  groups  that 
are  comparable.  Even  for  an  identical  clinical  question,  randomized  clinical  trials  will  differ  in  some 
subtle  and  some  not-so-subtle  ways,  and  it  is  necessary  to  determine  which  studies  are  sufficiently 
aimjlar  SO  that  a  qualitatively  or  quantitatively  pooled  result  has  meaning.  The  determination  of 
clinical  sensibility  is  primarily  a  chnical  task  and  requires  input  firom  experienced  clinicians.  This 

18  The  quality  assessment  of  studies  and  the  organization  of  those  studies  into  homogeneous  subgroups 
are  two  different  processes. 
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issue  of  dealing  with  ^'clinical  heterogeneity” — ensuring  that  a  literature  review  is  not  criticized  for 
“combining  apples  with  oranges” — cannot  be  underestimated  (Mulrow,  Langhorne,  and  Grimshaw, 
1997). 

Finally,  the  evidence  is  summarized,  often  in  evidence  tables  of  studies  and  characteristics. 
If  the  studies  are  few  or  heterogeneous,  the  summary  may  be  restricted  to  qualitative  conclusions, 
such  as  “the  superiority  of  the  new  method  has  not  been  estabhshed”  or  “formal  economic  analyses 
are  not  available,  but  the  new  drug  will  enable  treatment  to  be  done  on  an  outpatient  basis  rather 
than  in  the  hospital.”  Otherwise,  applying  statistical  pooling  or  meta-analysis  techniques  will 
generally  provide  the  most  information.  Meta-analysis  aggregates  the  results  of  multiple  studies, 
thereby  increasing  sample  size  and  allowing  more  statistically  powerful  conclusions  regarding  a 
common  treatment  or  intervention  effect  across  studies.  However,  meta-analysis  cannot  overcome 
poor  study  quaUty  and  can  produce  deceptive  results  when  based  on  poor-quality  data.  Qualitative 
systematic  reviews  pose  httle  risk  in  this  regard. 

Expert  Opinion.  Using  formal  approaches  to  obtain  expert  opinion  generally  involves 
capturing  and  synthesizing  the  opinion  of  a  broad-based,  unbiased  panel  informed  by  objective 
information  representing  all  sides  of  an  issue  (Brook,  1994).  The  information  exchange  usually 
begins  with  materials  delivered  to  the  panel  and  usually  includes  a  face-to-face  meeting,  which  may 
be  public  (Kanouse  et  al.,  1989),  Most  often,  the  evaluators  will  also  organize  some  form  of 
interaction  or  deliberations,  which  are  usually  structured  and/or  controlled  by  a  facilitator.  Finally, 
the  evaluator  should  use  an  exphcit  means  of  aggregating  judgments  to  identify  consensus  (Fink  et 
al.,  1984).  While  the  best  method  of  aggregation  is  not  clear,  mathematical  approaches  combining 
panelists’  scores  have  the  advantage  of  being  quantitatively  reportable,  and  thus  are  preferable  to 
non-quantitative  approaches.  The  scoring  process  is  often  iterative,  with  feedback  of  combined 
scores  to  participants  and  sometimes  further  discussions  prior  to  re-scoring  (Brook  et  al.,  1986). 

High-quality  expert  assessments  can  be  resource -intensive:  preparing  materials,  selecting 
the  panel,  and  choosing  methods  for  scoring  and  reporting  can  all  be  substantial  undertakings.  In 
some  cases,  these  costs  can  be  avoided  by  taking  advantage  of  the  private  or  public  efforts  of  others, 
as  long  as  the  implications  of  the  composition  of  the  panel,  the  completeness  and  timehness  of  the 
information  considered,  and  the  approach  to  aggregating  panel  judgments  are  understood. 
Alternatively,  a  hmited  effort  could  include  the  more  important  elements  of  a  formal  process  by 
choosing  a  few  (more  than  one  or  two)  experts,  giving  them  the  evidence  in  advance,  and  ensuring 
that  all  participate  in  generating  a  consensus. 
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Mathematical  Assessment.  A  decisionmaker  may  choose  to  utihze  mathematical 
assessments  instead  of,  or  in  addition  to,  formal  expert  opinion  and  evidence  assessment.  Several 
decisions  must  be  made  in  performing  a  mathematical  analysis  to  quantify  the  relationship 
between  the  costs  and  benefits  of  a  policy  (Haddix  et  al.,  1996): 


•  Define  the  Perspective  of  the  Analysis.  In  the  context  of  this  report,  the  perspective  of  the  analysis 
will  normally  be  the  health  plan.  However,  for  certain  applications,  other  perspectives  may  be 
included:  for  example,  the  perspective  of  the  employer-purchaser  in  including  lost  productivity  or 
the  perspective  of  society  in  including  remote  health  effects  (i.e.,  after  the  patient  changes  plans 
or  jobs)  for  a  government  purchaser  or  when  attempting  to  characterize  all  likely  social  impacts. 

•  Define  the  Time  Frame  of  Concern.  The  choice  of  time  frame  generally  includes  all  points  at  which 
relevant  costs  or  benefits  might  accrue.  For  example,  some  benefits  of  a  drug  that  prevents 
pregnancy  cannot  be  reahzed  for  at  least  nine  months  after  its  introduction. 

•  Define  the  Coverage  Options  to  Be  Compared.  The  choice  of  decision  options  (and,  hence,  coverage 
options)  to  be  compared  is  critical  to  the  usefulness  of  the  model.  For  example,  an  analysis 
comparing  the  candidate  treatment  to  a  poHcy  of  no  treatment  will  be  of  httle  practical  use  when 
some  level  of  treatment  is  the  status  quo  standard  of  care  (Drummond  et  al.,  1997). 

•  Define  and  Estimate  the  Costs  of  Each  Option.  When  defining  and  estimating  costs,  an  accurate 
result  depends  on  considering  aU  relevant  direct  and  indirect  costs  (Luce  and  Ehxhauser, 

1990).20 

•  Define  and  Estimate  the  Benefits  of  Each  Option.  To  define  and  estimate  a  benefit  requires 
additional  data.  Cost-effectiveness  analyses  often  rely  on  simple  chnical  measures,^^  For  cost- 
utility  analyses,  one  has  to  address  the  more  daunting  task  of  estimating  the  utility  (e.g.,  value) 
of  different  health  states  and  length  of  hfe,  but  the  outcomes  are  generahzable  across  different 
types  of  interventions.  Methods  for  utihty  estimation  include  direct  estimation  using  techniques 
such  as  standard  gamble,  time  trade-off,  rating  scales,  and  wiDingness-to-pay  or  indirect 
measurement  from  measures  of  health-related  quahty  of  life  such  as  the  quality  of  well-being 


Some  expert  opinion  and/or  evidence  assessment  is  necessary  to  build  the  models,  but  decisionmakers 
may  either  choose  not  to  or  not  be  able  to  carry  these  assessments  to  the  point  at  which  they  directly 
affect  decisionmaking. 

20  Our  colleague  Steven  Garber  points  out  that  making  decisions  for  society  that  favor  the  rich  might  be 
avoided  by  defining  value  for  all  persons  according  to  the  wiUingness-to-pay  of  persons  with  the  median 
income.  However,  this  may  require  an  assumption  that  preferences  are  ordered  by  income,  and  may 
simply  be  moving  the  problem  down  the  income  scale. 

21  That  is,  for  cost-effectiveness  analysis  in  the  narrow  sense  used  in  this  report. 
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index  (Haddix  et  al.,  1996;  Gold  et  al.,  1996).22  Often,  the  evaluator  will  use  the  results  of  these 
utihty  assessments  to  weight  or  discount  life  expectancy  to  create  an  estimate  of  the  net  total 
utility,  generally  expressed  as  the  equivalent  in  years  of  perfect  health  or  quality-adjusted  life 
years  (QALYs).  For  instance,  if  hving  one  year  with  severe  chronic  pain  is  equivalent  to  living 
six  months  with  no  chronic  pain,  then  one  year  of  Hving  with  severe  chronic  pain  has  the  value  of 
0.5  quahty-adjusted  life  year.  However,  the  evaluator  can  choose  to  use  other  similar  (e.g., 
healthy -year  equivalents)  or  simpler  (e.g.,  disability-free  years)  units.  For  cost-benefit  analyses, 
the  evaluator  has  to  take  the  additional  step  of  converting  outcomes  to  dollar  equivalents.  The 
available  approaches  include  the  human  capital  approach  of  valuing  a  year  of  life  saved  as  the 
net  present  value  of  a  person’s  economic  output  during  that  year  or  calculating  a  value  from  the 
premium  paid  to  workers  in  risky  occupations.  All  of  these  approaches  are  controversial  in 
medical  applications  for  a  variety  of  reasons,  including  perceived  equity  issues  of  valuing 
different  individuals’  fives  differently. 

•  Perform  Sensitivity  Aiiolysis.  Because  the  result  of  a  cost-outcome  analysis  is  a  single  ratio, 
traditional  statistical  significance  tests  cannot  be  used  to  quantify  uncertainty.  Rather, 
evaluators  quantify  the  impact  of  uncertainty  through  sensitivity  analysis,  which  involves  varying 
the  model  assumptions  (e.g.,  percent  of  antibiotic  failures  with  oral  treatment  for  a  specified 
condition)  over  a  reasonable  range  in  order  to  illustrate  the  effects  of  uncertainty  on  the  results 
of  the  analysis.  One’s  confidence  in  the  model’s  conclusions  should  depend  on  how  well  they 
hold  up  to  reasonable  variations  in  these  inputs. 


As  mentioned  in  the  preceding  subsection,  “Mathematical  Models,”  the  common  final  step  of 
a  cost-outcome  analysis  is  interpreting  the  resulting  cost-to-benefit  ratio.  For  example,  Figure  3.2 
illustrates  the  nine  categories  into  which  strategies  may  fall  when  they  are  evaluated  by  the  criteria 
of  incremental  costs  and  incremental  effectiveness  over  an  alternative.  It  is  a  modified  version  of  the 
matrix  of  costs  and  benefits  devised  by  O’Brien  et  al.  (1997).  Three  cells  (labeled  Accept) 
characterize  strategies  that  are  at  least  as  effective  and  no  more  costly  than  the  point  of  comparison 
and  should  be  adopted.  Similarly,  three  cells  (labeled  Reject)  characterize  strategies  that  are  no 
more  effective  and  at  least  as  costly  and  therefore  should  be  rejected.  However,  the  optimal  course 
of  action  for  strategies  falling  in  the  remaining  three  cells  (labeled  Judgment)  is  unclear,  because 


22  None  of  these  methods  is  a  gold  standard.  For  instance,  equating  the  value  of  an  outcome  with  how 
much  someone  would  pay  to  achieve  it  (willingness-to-pay)  may  create  a  bias  toward  outcomes  favored  by 
the  wealthy  (Drummond  et  al.,  1997).  Similarly,  asking  persons  to  trade  quality  for  time  alive  may  create 
a  bias  toward  outcomes  favored  by  healthier  people,  who  anticipate  having  more  healthy  fife-years  left 
(Drummond  et  al.,  1997). 
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Figure  3.2— Cost-Effectiveness  Grid 

these  cells  indicate  that  strategies  are  less  effective  and  less  costly,  similarly  effective  and  similarly 
costly,  or  more  effective  and  more  costly. 

A  fuller  analysis  may  illuminate  the  correct  decision.  Often,  however,  decisions  will  have  to 
be  made  on  other  grounds  or,  for  more  expensive  and  more  effective  interventions,  be  based  on 
whether  the  strategy  represents  a  reasonable  investment.  However,  no  set  threshold  exists  for  the 
ratio  at  which  a  strategy  can  be  recommended.  Some  authors  have  asserted  that  strategies  costing 
less  than  $20,000/QALY  (in  1992  Canadian  dollars)  are  quite  cost-effective,  that  strategies  costing 
between  $20,000  and  $100,000/QALY  are  reasonable,  and  that  those  costing  over  $100,000  are 
suspect  (Laupacis  et  al.,  1992).  It  is  sometimes  possible,  and  usually  useful,  to  place  the 
recommendation  in  context  by  including  a  league  table,  which  shows  similarly  constructed  cost- 
outcomes  assessments  for  other  interventions  that  have  and  have  not  been  adopted  by  this  or  other 
organizations,^  particularly  after  an  organization  has  a  history  of  conducting  such  analyses. 
Including  one’s  own  analysis  in  the  table  allows  for  ready  comparisons  and  acts  as  a  check  for 
internal  consistency  in  analysis  and  decisionmaking. 

Decision  Analysis.  To  obtain  estimates  of  effectiveness  for  cost-outcome  analysis  or  for 
direct  comparison  of  strategies,  evaluators  often  use  decision  analysis.  The  steps  in  a  decision 
analysis  are  as  follows:  (1)  define  the  strategies  imder  consideration  and  the  time  horizon,  (2)  define 
the  model  according  to  the  strategies  and  time  fi-ame,  (3)  translate  the  consequences  into  a  relevant 
metric,  and  (4)  perform  sensitivity  analysis.  The  first,  third,  and  fourth  steps  are  similar  to  those 
for  cost-outcome  analysis;  these  steps  are  not  described  here.  The  second  step  involves  defining  the 
appropriate  model.  As  noted  above,  the  model  usually  used  in  decision  analysis  is  the  decision  tree, 
because  it  provides  an  expUcit  graphical  explanation  of  trade-offs  between  the  risks  and  benefits  of 
different  strategies,  thereby  justifying  coverage  decisions.  Figure  3.3  illustrates  this  point  with  a 
simple  example  of  a  decision  tree. 

The  analysis  of  the  tree  follows  a  patient  from  left  to  right.  At  each  node,  an  event  occurs 
(i.e.,  either  a  decision  or  a  chance  happening)  that  sends  the  patient  along  a  branch  to  the  right. 


23  The  term  “league  table”  originated  in  tables  of  British  soccer-league  standings. 
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For  instance,  the  issue  may  be  whether  to  cover  beta-blockers  after  a  heart  attack.  If  the 
organization  covers  beta-blockers,  this  model  indicates  that  one  of  two  events  will  occur  with  some 
probability:  the  patient  will  both  be  eligible  to  take  beta-blockers  (that  is,  have  chnical  indications 
for  taking  them)  and  take  the  beta-blockers,  or  not.  If  the  patient  does  not  take  beta-blockers,  then 
he  or  she  has  some  increased  probability  of  experiencing  the  outcome  of  interest,  another  heart 
attack.  Using  this  tree,  the  evaluator  “works  back”  from  the  outcomes  at  the  far  right  to  calculate 
the  relative  value  of  each  represented  decision  strategy  from  the  probabihties  along  each  path.  The 
expected  result  of  each  strategy  is  the  weighted  sum  of  all  the  possible  outcomes  at  the  end  of  all 
possible  paths  for  that  strategy;  the  weight  for  each  outcome  is  equal  to  the  overall  probability  of 
that  outcome  as  defined  by  the  tree. 


In  the  beta-blocker  example,  the  outcome  could  simply  be  the  probability  of  a  second  heart 
attack,  or  it  could  be  converted  into  a  metric  such  as  survival.  When  the  same  decision  is  made 
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repeatedly  over  time — say,  to  consider  third  as  well  as  second  heart  attacks — the  model  can  be 
modified  to  be  iterative,  or  what  is  known  as  a  Markov  process  model  (Pauker  and  Kassirer,  1987), 

Demand  Assessments.  Standards  for  assessing  the  chnical  demand  for,  or  actual  use  of, 
pharmaceuticals  are  less  well  developed  than  decision  analyses.  For  example,  in  the  decision  tree 
above,  demand  assessment  woxild  include  estimation  of  the  likehhood  that  eligible  patients  are 
prescribed,  and  take,  beta-blockers,  as  well  as  the  more  subtle  possibihty  that  use  of  a 
pharmaceutical  may  lead  to  the  identification  of  more  candidates.  In  the  past,  simple  models  based 
on  disease -prevalence  and  clinical  criteria  were  the  standard  for  predicting  use,  and  such  models  are 
still  appropriate  for  rough  estimation.  However,  this  approach  has  failed  historically  in  determining 
physician  manpower  and  hospital-bed  requirements,  and  more  recently  has  failed  to  predict  the  use 
of  “lifestyle  drugs”  and  drugs  that  are  advertised  directly  to  consumers  (Feldstein,  1988).  Clearly, 
serious  demand-assessment  efforts  need  to  be  more  sophisticated. 

One  approach  to  formulating  models  that  are  more  adequate  is  to  conceptualize 
pharmaceutical  demand  in  terms  of  stocks  and  flows.  Stocks  are  the  number  of  users  of  the  drug  in 
question  among  the  population  of  covered  lives;  flows  are  any  additions  to  or  subtractions  fi'om  the 
number  of  users  during  a  given  period.  Seen  in  this  way,  use  of  a  particular  drug  diming  any  period 
of  time  can  be  calculated  from  three  pieces  of  information:  (1)  the  number  of  people  on  treatment 
within  a  given  period  of  time  (the  “stock”  in  the  prior  period  plus  the  net  flow  [change  in  number  of 
users]),  (2)  dosage,  and  (3)  duration  of  use.  Thus,  demand  assessment  is  complex  because  of  the 
need  to  account  for  all  of  the  components  of  “reaUzed  demand,”  or  pharmaceutical  use  (initiation, 
dosage,  and  duration). 

Steps  Common  to  All  Models.  Although  the  steps  in  constructing  a  model  can  vary,  four 
factors  are  nearly  always  relevant. 

The  first  is  identifying  the  decisionmakers  whose  choices  influence  demand.  These 
decisionmakers  can  be  one  of  four  sets  of  actors:  (1)  physicians  only,  the  key  actors  in  emergent 
conditions;  (2)  physicians  and  patients,  when  choice  of  treatment  is  more  discretionary;  (3)  patients 
only,  when  comphance  and  duration  are  being  considered;  and  (4)  pharmacists,  when  issues  of 
substitution  and  compliance  with  drug-management  programs  (e.g.,  disease-management,  step- 
therapy  protocols)  are  key  considerations. 

The  second  issue  is  specifying  those  choices  that  are  actually  to  be  modeled.  Examples 
include  the  patient  s  decision  to  seek  care  for  a  condition  or  the  physician’s  decision  to  prescribe  a 
particular  drug. 
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The  third  issue  is  defining  the  set  of  alternatives  that  the  decisionmakers  face.  For 
example,  if  there  is  interest  in  the  decision  to  initiate  drug  therapy,  the  choice  set  may  be  no 
treatment  and  treating  with  currently  available  drugs. 

The  fourth  issue  is  determining  the  characteristics  of  the  decisionmakers  and  the  attributes 
of  the  choice-set  factors  that  cause  them  to  choose  one  alternative  over  the  others  (Train,  1986). 
Examples  include  the  demographic  and  clinical  characteristics  of  the  patients  and  the  specialty  of 
the  physicians. 

At  this  point,  the  goal  is  to  formulate  the  impact  of  the  control  mechanism  on  the 
demand — for  example,  the  extent  to  which  prior-authorization  requirements  for  certain  drugs 
discourage  use  by  increasing  time  requirements  and  other  difficulties  in  prescribing.  A  demand 
model  will  be  most  useful  when  this  factor  can  be  ascertained.  If  it  cannot,  the  model  can  be  used 
to  estimate  use  but  not  how  organizational  choices  influence  use. 

Make  a  Decision 

The  steps  in  the  process  outlined  above  can  inform  a  decision,  but  they  cannot  make  a 
decision.  It  is  usually  too  difficult  for  a  single  analysis  to  incorporate  all  relevant  considerations, 
such  as  the  administrative  burden  of  adopting  a  new  pharmaceutical  or  an  organizational  interest 
in  pursuing  a  particular  strategic  behavior.  Even  when  a  model  includes  all  of  the  important 
factors,  it  may  not  point  to  a  clear  decision.  For  example,  the  model  may  predict  a  number  of 
dissimilar  outcomes  that  it  can  be  difficult  to  aggregate  into  an  overall  summary  score. 

In  such  cases,  judgment  must  be  used  to  determine  the  relative  importance  of  each  outcome 
(e.g.,  if  promoting  use  of  a  drug  may  be  predicted  to  result  in  increases  in  both  appropriate  and 
inappropriate  use,  which  is  more  important?).  Moreover,  instituting  innovations  may  have  strategic 
implications  for  the  organization  that  may  mean  that  existing  practices  will  have  to  be  disrupted, 
affecting  provider  morale,  patient  convenience,  etc.  Therefore,  deciding  to  accept  or  reject  a  new 
pharmaceutical  technology  can  raise  non-analytical  questions  of  organizational  and  social  values, 
such  as  how  much  should  be  invested  to  obtain  a  particular  outcome  and  what  technologies  or 
services  should  be  sacrificed  to  obtain  that  outcome. 

In  the  next  section,  we  look  at  how  to  apply  the  framework  described  above.  Both  thorough 
and  abbreviated  approaches  are  presented  to  ensure  that  the  most  important  information  is 
obtained  on  each  product. 
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4.  Applying  the  Framework  Practically 

The  decisionmaking  framework  described  in  Section  3  provides  a  “road  map”  for 
pharmaceutical  technology  assessments.  It  is  not  an  unvarying  recipe  for  how  to  make  decisions 
about  pharmaceutical  technologies.  Each  organization  must  tailor  its  approach  according  to  its  own 
priorities  and  the  information  and  resources  that  are  available.  However,  consistent  apphcation  of 
this  framework  (or  a  modified  version  of  it)  will  expose  hidden  biases,  reduce  the  risk  of  illogical  or 
poorly  thought-through  decisions,  and  position  the  organization  to  reap  the  rewards  of  responding  to 
patient  and  purchaser  needs  in  a  fiscally  and  medically  responsible  way.  Use  of  a  comprehensive 
framework  will  achieve  these  benefits  by  providing  an  objective  and  repeatable  process  for  analyzing 
promising  new  pharmaceuticals.  Compared  to  less  structured  approaches,  the  results  should  be 
more  defensible  to  internal  and  external  chents,  less  dependent  on  the  involvement  of  particular 
individuals,  and  better  able  to  support  comparisons  across  specific  situations  and  time  periods. 

Decisionmakers  hardly  need  to  be  reminded  of  the  barriers  in  applying  this  framework  to 
real-world  situations  (Matuszewski,  1997): 

•  a  failvire  to  ahgn  organizational  incentives  to  do  an  evaluation 

•  lack  of  evidence  or  inconsistent  evidence  upon  which  to  base  analyses 

•  lack  of  agreement  on  how  to  perform  an  assessment  \mder  a  particular  circumstance 

•  inadequate  financial  or  human  resources. 

The  rational  yet  practical  decisionmaker  must  devise  strategies  for  working  within  these 
confines.  Thus,  a  central  task  for  the  decisionmaker  is  to  determine  which  evaluations  and  which 
steps  of  those  evaluations  need  greater  or  lesser  attention.  In  some  circumstances,  the  lack  of 
human  resources  may  lead  the  decisionmaker  to  buy  (rather  than  make)  an  analysis,  or  to  improve 
local  capabilities.  In  others,  the  decisionmaker  must  devise  a  resource-sparing  means  of 
implementing  a  structured  evaluation  process. 

A  first  step  to  evaluation  is  realizing  that,  in  the  context  of  an  ongoing  technology 
assessment  effort,  many  of  the  tasks  outlined  in  Section  3  need  not  be  resource-intensive.  For 
example,  if  the  characteristics  of  the  decisionmaking  organization  are  fairly  static,  they  need  not  be 
regenerated  with  each  new  assessment.  Rather,  the  evaluator  could  make  what  could  be 
essentially  a  one-time  effort  to  create  a  data  file  to  address  the  issues  outlined  in  Table  3.2,  relying 
largely  on  internal  information.  Alternatively,  a  more  extensive  but  seldom  necessary  approach 
would  involve  profiling  alternative  organizational  structures.  In  any  case,  with  periodic  updates, 
this  generic  file  could  contribute  to  the  assessment  process  for  many  drugs. 

Similarly,  by  drawing  from  the  organization’s  strategy  and  philosophy  toward  drug 

management  in  general,  the  decisionmaker  or  evaluator  could  identify  a  generic  list  of  decision 
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options.  For  most  drugs,  the  decisionmaker  would  then  need  to  consider  only  the  existing  list  or 
small  amendments  to  it.  However,  decisionmakers  should  always  be  alert  to  the  possibility  that 
they  are  considering  an  important  drug  (i.e.,  those  true  advances  with  the  potential  to  increase  the 
effectiveness  or  efficiency  of  care,  or  to  create  new  classes  of  care)  that  may  merit  modification  of 
existing  rules,  even  if  such  modification  requires  shifting  procedures  or  benefit  structures. 

Characterizing  the  drug  also  need  not  be  resource-intensive  in  all  situations.  A  first  cut  at 
the  issues  surrounding  the  drug  might  be  derived  from  the  manufacturer  s  publications,  FDA 
reports,  peer-reviewed  articles  including  formal  or  informal  reviews,  and  the  decisionmaker’s  own 
experience  with  the  manufacturer  and  similar  drugs.  Because  this  step  calls  for  the  identification  of 
issues  rather  than  a  formal  evaluation  of  evidence,  a  documented  but  relatively  simple  search  for 
information  may  suffice.  Since  the  evidence  gathering  that  takes  place  throughout  the  process  will 
provide  further  information  about  the  drug,  that  information  can  be  used  to  augment  the 
characterization  of  the  drug  subsequently. 

Choosing  the  ty^e  and  level  of  analysis  needed  to  make  a  decision  will  also  be  subject  to 
constraints  on  time,  resources,  and  skills.  Because  the  results  of  one  lead  naturally  to  the  next, 
evidence  assessment  should  be  considered  first,  then  expert  assessment,  and,  finally,  mathematical 
assessment.  Evidence  assessments  will  be  most  useful  and  possibly  sufficient  when  considering  a 
focused  clinical  question  for  which  there  are  at  least  several  good  studies.  When  the  hterature  does 
not  provide  adequate  information,  as  is  often  the  case,  the  decisionmaker  should  seek  to  extend  the 
usefulness  of  existing  information  and  experience  through  expert  or  mathematical  assessment. 
Which  of  these  two  alternative  methods  to  use  depends  upon  the  amount  of  time  and  staff 
available,  and  upon  the  recognition  that  the  resulting  information  for  decisionmaking  will  vary 
greatly  according  to  the  choice  made  (e.g.,  cost/life  year  versus  greater  than  50%  agreement  by 
experts).  Expert  assessments  can  be  faster,  but  are  arguably  less  exphcit  and  do  not  generally 
provide  quantitative  results.  Consider  a  situation  in  which  the  hterature  indicates  that  the  drug 
has  higher  efficacy  and  costs  less  than  its  substitute  in  certain  circumstances,  but  does  not  provide 
pertinent  information  about  the  values  of  these  quantities.  In  this  case,  an  expert  assessment  may 
be  appropriate  but  a  cost-outcome  assessment  may  be  more  useful. 

Although  performing  a  full  evidence  assessment  may  not  be  necessary  or  possible  in  a  given 
situation,  the  underlying  principles  discussed  in  Section  3  should  guide  the  process.  For  example, 
evidence  assessment  should  at  least  include  information  from  the  authoritative  Cochrane 
Collaboration  database  or  other  sources  for  systematic  reviews.  If  an  adequate  evidence  report  is 
not  available,  it  is  often  possible  to  perform  systematic  searches  of  literature  databases  (as  opposed 
to  convenience  searches),  and  at  least  some  assessment  of  the  strength  of  the  evidence  uncovered 
(preferably  by  midtiple  readers).  When  an  expert  assessment  is  appropriate  and  there  are  few  or  no 
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recent  studies  addressing  the  specific  question  at  hand,  the  decisionmaker  should,  at  a  minimum, 
seek  opinions  from  multiple  experts  with  diverse  backgrounds  and  perspectives,  ideally  after 
allowing  for  a  review  of  the  evidence  and  an  informed  discussion  among  them. 

Similarly,  while  mathematical  models  are  best  developed  using  formal  methods  with 
sensitivity  analyses,  hmited  (“back  of  the  envelope”)  calculations  can  provide  important  insights. 

For  example,  parameter  estimates  and  results  arising  from  such  an  approach  may  be  so  clear  as  to 
make  more-complete  analysis  very  unlikely  to  move  a  new  treatment  out  of  the  Accept  or  Reject 
region  of  Figure  3.2.  In  all  cases,  such  an  exercise  provides  guidance  on  the  level  of  rigor  required  for 
a  more  formal  mathematical  assessment,  and  simply  creating  a  schematic  for  an  exphcit 
mathematical  model  (e.g.,  a  decision  tree)  can  be  useful  in  helping  to  identify  key  relationships  and 
in  providing  grounds  for  discussion  between  experts  (Roberts  and  Edein,  1984).  Moreover,  the  shell 
of  a  model  might  be  used  to  investigate  whether  the  optimal  pohcy  changes  under  extreme  values 
absent  the  intention  or  possibility  of  determining  best  estimates  for  parameters,. 

Finally,  while  it  may  be  appropriate  to  omit  some  steps  entirely  in  certain  circumstances, 
evaluators  will  want  to  consider  at  least  abbreviated  approaches  to  working  through  each  step  of 
the  framework.  For  reference.  Table  4.1  summarizes  both  thorough  and  abbreviated  approaches. 
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Table  4.1  ^Thorough  and  Abbreviated  Approaches  to  Technology  Assessment 


Approach  Type 

Step 

Thorough  Approach 

Abbreviated  Approach 

Evidence 

Assessment 

Collect  evidence 

With  the  assistance  of  a 
professional  librarian,  search 
multiple  library  databases  in 
all  languages,  as  well  as  the 
industry  literature,  and 
identify  unpublished  studies. 

Search  only  a  single  English 
database — e.g., 

Medline — for  a  limited  range 
of  publication  years. 

Sift  through 
evidence 

Develop  an  explicit, 
reproducible  coding  scheme 
for  rejection  and 
acceptance  of  literature  for 
inclusion  in  review,  and  for 
extraction  of  data.  Use 
multiple  readers. 

Allow  for  a  "miscellaneous" 
rejection  code.  Use  a  single 
reader  to  decide  on 
inclusion/exclusion  and  for 
data  extraction. 

Classify  evidence  by 
quality 

Develop  an  explicit  scheme 
for  valuing  evidence.  For 
example,  rank  evidence 
using  the  five  classifications 
of  the  Canadian  Task  Force 
system,  and  quality-score 
studies  using  a  validated 
instrument.  Use  multiple 
reviewers. 

Rank  evidence  using  five  or 
fewer  levels  (e.g., 

Excellent,  Good,  Fair)  or  do 
not  quality-score  studies. 

Use  a  single  reviewer. 

Summarize 

evidence 

Perform  a  meta-analysis  and 
write  a  full  report  including 
evidence  tables. 

Write  an  abbreviated  report 
or  give  an  oral  presentation 
(e.g.,  at  the  quarterly  P&T 
meeting)  without  meta- 
anaiysis — i.e.,  perform  a 
qualitative  analysis  only. 

Expert  Assessment 

Select  experts 

Select  a  diverse  set  of  about 
twelve  experts  from 
multiple  disciplines  and 
professional  affiliations. 

Maintain  a  list  of  contacts  in 
various  fields  and  select  at 
least  three  experts  who 
represent  only  a  few 
disciplines  or  affiliations. 

Provide  background 
to  experts 

Perform  a  literature  review 
(or  full  evidence  assessment) 
and  provide  to  experts 
before  soliciting  opinion. 

Give  a  brief  presentation  at 
beginning  of  face-to-face  (or 
teleconference)  meeting.  If 
no  meeting,  provide 
abbreviated  material  before 
soliciting  opinion. 

Solicit  opinion 

Collect  written  responses 
and  distribute  a  summary  of 
responses  to  the  experts. 
Follow-up  with  a  mediated, 
face-to-face  discussion. 

Collect  only  written 
responses  or  collect  only 
face-to-face  responses  with 
no  follow-up. 

Aggregate 

judgments 

Use  an  explicit,  reproducible 
method  for  summarizing 
findings.  For  example, 
statistically  summarize 
responses  to  numeric 
questions. 

Provide  narrative  of  the 
responses. 
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Table  4.1 — continued 

Approach  Type 

step 

Thorough  Approach 

Abbreviated  Approach 

II  Mathematical 

II  Assessment 

Cost-Outcome  and 
Decision  Analysis 

Define  perspective 

Use  the  health  plan’s 
perspective. 

Take  the  perspective 
required  by  the  choices 
made  below.  For  example, 
estimates  of  cost  from  the 
literature  usually  take  the 
societal  perspective. 

Define  time  frame 
of  concern 

Consider  the  entire  horizon  of 
possible  effects  (including 
secondary  or  indirect 
effects). 

Use  best  guesses  about 
downstream  effects. 

Define  coverage 
options  to  be 
compared 

Accurately  characterize  the 
full  range  of  decision  options. 

Develop  stylized 
descriptions  of  the  most- 
promising  and  estimable 
options. 

Define  and 
estimate  costs  of 
each  option  (for 
Cost-Outcome 
only) 

Include  both  direct  and 
indirect  costs  (including 
downstream  costs),  with 
estimates  based  on  an 
evidence  assessment  or 
reliable  internal  information 
where  available. 

Estimate  direct,  immediate 
costs  based  on  informal 
evidence  or  expert  review 
or  conjecture.  May  rely  on 
best  guesses  about  indirect 
and  downstream  costs. 

Define  and 
estimate  benefits 
of  each  option 

Include  both  direct  and 
indirect  benefits  (including 
downstream  benefits)  with 
estimates  based  on  an 
evidence  assessment  or 
reliable  internal  information 
where  available. 

Estimate  direct,  immediate 
benefits  based  on  informal 
evidence  or  expert  review 
or  conjecture.  May  rely  on 
best  guesses  about  indirect 
and  downstream  benefits. 

Perform  sensitivity 
analysis 

Fully  characterize  the 
robustness  of  the  results  by 
varying  the  above  estimates 
over  any  conceivably 
relevant  range. 

Explore  the  robustness  of  the 
results  to  only  a  few 
interesting  alternative 
estimates. 
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Table  4.1 — continued 


Approach  Type 

Step 

Thorough  Approach 

Abbreviated  Approach 

Demand  Analysis 

Identify  relevant 

Consider  entire  sequence  of 

Limit  focus  to  one  or  two  key 

decisionmakers 

decisions  involved  in 

actors — usually,  physicians 

and  corresponding 

determining  initiation  and 

and  patients — using  best 

decisions 

duration  of  use. 

guesses  about  the  influence 
of  other  actors  on  use. 

Characterize 

Obtain  information  from 

Rely  on  secondary  data 

decision  elements 

medical  records, 

sources,  interviews,  or  focus 

•  Universe  of 

supplemented  with 

groups  with  patients  and 

alternative 

treatment 

epidemiological  data, 
representative  surveys  of 

physicians. 

•  Key 

patients  and  physicians,  and 

Formulate  “best  guesses" 

decisionmakers 

existing  literature  where 

about  effect  of  unmeasured 

•  Relevant  drug 
attributes 

•  Decisionmaker 
characteristics 

appropriate. 

elements  on  use. 

Incorporate 

Specify  and  estimate 

Use  informally  obtained 

elements  into 

econometric  models  with 

information  combined  with 

demand  models 

patient-level  data.  Use 
resulting  parameters  in 
predicting  impact  of  new 
drug.  Validate  predictions 
through  comparisons  with 
take-up  rates”  resulting  from 
earlier  introductions  of  new 
drugs. 

“best  guesses”  to  create 
back-ot-the-envelope 
models  under  alternative 
sets  of  assumptions.  Use 
earlier  introductions  of 
related  drugs  to  gauge 
effects. 

^Take-up  rates  are  how  quickly  consumers  and  providers  adopt  a  new  drug. 


Summary 

In  summary,  we  have  provided  a  decisionmaking  framework  in  response  to  the  need  for 
better  evaluations  of  new  pharmaceutical  technologies  by  managed  care  organizations.  Use  of  the 
framework  will  not  create  evidence.  Instead,  it  should  stimulate  decision  analyses  that  are  broader, 
more  explicit,  more  efficient,  more  reproducible,  and  more  relevant  than  those  now  being  performed. 
Similarly,  while  it  will  not  increase  resources,  it  should  stimulate  skill-building  and  an  explicit 
discussion  of  the  trade-offs  between  the  rigor  of  an  evaluation  and  budgetary  and  data  constraints. 
Finally,  we  hope  that  it  will  assist  in  implementing  a  systematic  analytic  process  for 
pharmaceutical  technology  assessment  that  should  ultimately  improve  the  health  of  the  populations 
under  care  and  the  effectiveness  of  the  organizations  that  serve  them. 
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Appendix  A:  Annotated  Bibliography 
Pharmaceuticals  in  Healthcare 


Schulman  KA,  Rubeinstein  LE,  Abernethy  DR,  Seiis  DM,  Sulmasy  DP.  The  effect 
of  pharmaceutical  benefits  managers:  is  it  being  evaluated?  Ann  Intern  Med.  May 
1996;124(10):906-913. 

The  authors  summarize  the  history  and  current  role  of  pharmaceutical  benefits  managers,  and  identify  their 
contribution  to  health  care  delivery,  as  well  as  their  limitations. 


Schweitzer  SO.  Pharmaceutical  Economics  and  Policy.  New  York,  NY:  Oxford 
University  Press;  1997. 

An  overview  of  the  economics  of  drug  manufacturing,  use,  promotion,  and  regulation  in  the  United  States 
and  abroad,  this  book  contains  an  interesting  discussion  of  the  forces  driving  increasing  expenditures  on 
pharmaceuticals  and  of  the  role  of  pharmaceuticals  in  the  health  production  process  from  an  economic 
perspective. 


Current  Practices 


Dubinsky  M,  Ferguson  JH.  Analysis  of  the  National  Institutes  of  Health  Medicare 
coverage  assessment.  Int  J  Technol  Assess  Health  Care.  1990;6:480-488. 

The  authors  report  the  results  of  a  study  designed  to  measure  the  level  of  agreement  between  the  assessment 
performed  by  the  National  Institutes  of  Health  (NIH)  and  that  of  the  Office  of  Health  Technology 
Assessment  (OHTA).  Because  OHTA  adopts  NIH  conclusions  in  93%  of  the  cases,  it  appears  that  a 
significant  proportion  of  coverage  decisions  at  the  national  level  rely  on  expert  opinion  rather  than  on 
scientific  evidence. 

Luce  BR,  Brown  RE.  The  use  of  technology  assessment  by  hospitals,  health 
maintenance  organizations,  and  third-party  payers  in  the  United  States.  Int  J  Technol 
Assess  Health  Care.  1995;ll(l):79-92. 

Through  a  series  of  structured  in-depth  interviews  with  various  decisionmakers,  this  survey  attempts  to 
determine  the  reliance  on  technology  assessment  in  hospitals,  health  maintenance  organizations  (HMOs), 
and  third-party  payers. 

Lyles  A,  Luce  BR,  Rentz  AM.  Managed  care  pharmacy,  socioeconomic  assessments 
and  drug  adoption  decisions.  Soc  Sci  Med.  1997;45(4):511-521. 

The  auAors  report  on  a  telephone  survey  of  50  managed  care  plans  in  the  United  States.  The  survey 
determined  the  plan’s  formulary  type,  use  of  assessment  methodologies,  the  drug  coverage  approval  process, 
the  decisionmakers  within  the  organization,  the  sources  used  in  the  assessments,  and  the  decisionmakers’ 
attitude  toward  assessments. 
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Mather  DB,  Sullivan  SD,  Augenstein  D,  Fullerton  DS,  Atherly  D.  Incorporating 
clinical  outcomes  and  economic  consequences  into  drug  formulary  decisions:  a 
practical  approach.  Am  J  Managed  Care,  1999;5(3):  277-285. 

This  article  describes  an  evidence-based  formulary  adoption  process  used  by  a  large  managed  care 
organization.  The  process  is  unique  in  that  it  requires  drug  manufacturers  to  compile  and  submit  clinical 
and  economic  data  from  published  and  unpublished  sources  prior  to  review  and  to  create  models  that  the  plan 
can  use  to  forecast  health  outcomes  and  costs  of  care  associated  with  use  of  the  drug. 

Steiner  C,  Powe  NR,  Anderson  GF,  Das  A.  The  review  process  used  by  U.S.  Health 
care  plans  to  evaluate  new  medical  technology  for  coverage.  JGIM,  May 
1996;11:294-302. 

The  authors  conducted  a  survey  of  231  physicians  at  private  health  plans.  The  sample  covered  66%  and 
72%  of  the  HMO  and  indemnity  beneficiaries,  respectively,  in  the  United  States.  Tlie  survey  demonstrated 
that  final  coverage  decisions  for  medical  technology  are  not  uniform  from  plan  to  plan,  are  difficult  to 
predict,  and  differ  from  assessments  done  by  the  medical  scientific  community.  Different  health  plans  have 
different  review  processes,  confer  different  authority  on  the  medical  director,  have  different  timing  for 
making  decisions,  and  view  sources  and  evidence  differently. 

Steiner  CA,  Powe  NR,  Anderson  GF,  Das  A.  Technology  coverage  decisions  by 
health  care  plans  and  considerations  by  medical  directors.  Med  Care.  1997;35(5): 
472-489. 

This  survey  of  private  health  care  plans  in  the  United  States  demonstrates  considerable  variability  in 
whether  a  given  laser  technology  would  be  covered.  Clinical,  economic,  and  regulatory  issues  appear  to 
dominate  the  decisionmaking  processes  of  medical  directors. 

Weingart  SN.  Acquiring  advanced  technology:  decision-making  strategies  at  twelve 
medical  centers.  Int  J  Technol  Assess  Health  Care.  1993;9(4):530-538. 

This  paper  reports  on  a  survey  designed  to  describe  the  decisionmaking  strategies  regarding  the  acquisition 
of  technology  in  12  major  medical  centers.  Although  exemplary  models  exist,  the  technology  assessment 
process  at  most  institutions  is  described  as  “political,”  “informal,”  or  “ad  hoc.” 


Overview  of  Technology  Assessment 


Garber  AM.  Can  technology  assessment  control  health  spending?  Health  Aff, 

Summer  1994:115-126. 

Garber  summarizes  the  different  approaches  to  technology  assessment,  including  cost-effectiveness  analysis, 
appropriateness  evaluation  (a  formal  group  technique  to  elicit  expert  opinion),  and  evaluation  of  strength  of 
evidence  (requires  scientifically  rigorous  proof  of  effectiveness).  He  further  identifies  the  potential  barriers  to 
technology  assessment. 
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Goodman  CS.  TAlOl:  Introduction  to  Health  Care  Technology  Assessment, 
Washington,  D.C.:  National  Information  Center  on  Health  Services  Research  & 
Health  Care  Technology,  National  Library  of  Medicine;  January  1998. 
http://www.nlm.nih.gOv/nichsr/tal01/tal01  cl.htm 

This  report  contains  a  comprehensive  introduction  to  technology  assessment,  with  special  reference  to 
health  care. 

Matuszewski  KA.  A  primer  on  the  assessment  of  medical  technologies.  Pharm  Pract 
Manage  Q.  January  1997:53-65. 

This  article  covers  the  basics  of  technology  assessment,  including  what  it  is,  who  does  it,  how  it  is  done 
(perspectives,  essential  tools,  qualities  of  a  good  assessment).  The  author  also  notes  common  problems 
encountered  in  executing  a  technology  assessment. 


Choosing  What  to  Assess 


Eddy  D.  Selecting  technologies  for  assessment.  Int  J  Technol  Assess  Health  Care, 
1989;5:485-501. 

The  focus  of  this  article  is  on  setting  priorities  in  choosing  technologies  to  assess.  The  author  reviews 
the  qualitative  criteria  given  by  agencies  in  priority  setting.  He  suggests  a  quantitative  method  that  is 
essentially  a  decision  analytic  approach  to  priority  setting. 

Eddy  D.  A  Manual  on  Assessing  Health  Practices  and  Designing  Practice  Policies: 
The  Explicit  Approach.  Philadelphia,  Penn:  American  College  of  Physicians; 
1992. 

Provides  an  overview  of  the  author’s  approach  to  explicit  analysis  of  evidence,  estimation  of  outcome, 
calculation  of  costs,  and  assessment  of  preferences. 


Harper  G,  Townsend  J,  Buxton  M.  The  preliminary  economic  evaluation  of  health 
technologies  for  the  prioritization  of  health  technology  assessments:  a  discussion. 
Int  J  Technol  Assess  Health  Care.  1998;14(4):652-662. 

The  authors  review  the  existing  methods  for  priority  setting  in  technology  assessment  and  suggest 
elements  of  an  ideal  priority-setting  process. 


Evidence  Assessment 


Berlin  JA,  Antman  EM.  Advantages  and  limitations  of  meta-analytic  regressions  of 
clinical  trials  data.  Online  J  Curr  Clin  Trials.  4  June  1994;  Doc.  No.  124. 

This  publication  discusses  the  methodology  and  appropriateness  of  meta-regression,  which  has  become  an 
increasingly  common  technique  through  which  to  both  account  for  and  understand  heterogeneity  between 
studies  in  a  meta-analysis. 
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Berlin  JA.  Invited  commentary:  benefits  of  heterogeneity  in  meta-analysis  of  data 
from  epidemiologic  studies.  Am  J  Epidemiol,  995;142(4);383-387. 

This  author  defines  heterogeneity  in  the  meta-analysis  setting.  Generally,  heterogeneity  is  seen  as  an 
unfortunate  complication  in  a  meta-analysis  because  it  may  deter  the  analyst  from  combining  studies. 
Berlin  discusses  how  heterogeneity  benefits  a  meta-analysis. 


Borzak  S,  Ridker  PM.  Discordance  between  meta-analyses  and  large-scale  randomized, 
controlled  trials.  Ann  Intern  Med,  1995;123:873-877. 

This  article  discusses  the  strengths  and  weaknesses  of  meta-analysis.  The  authors  also  compare  meta- 
analytic  results  with  those  of  large  randomized-controlled  trials,  and  consider  the  settings  in  which  a  meta¬ 
analysis  may  not  be  definitive,  and  a  large  trial  is  necessary  to  answer  a  particular  clinical  question. 


The  Cochrane  Library,  The  Cochrane  Methods  Handbook,  1999;Issue  1.  Vista,  Calif.: 
Update  Software  (www.cochrane.org). 

The  Cochrane  Collaboration  is  an  international  nonprofit  organization  dedicated  to  “preparing,  maintaining 
and  promoting  the  accessibility  of  systematic  reviews  of  the  effects  of  health  care  interventions.”  The  first 
Center  was  established  in  Oxford,  England,  in  October  1992.  Since  then,  more  than  a  dozen  Centers  have 
opened  around  the  world.  In  particular,  the  Cochrane  Collaboration  maintains  a  library  of  systematic 
reviews  whose  abstracts  can  be  viewed  at  no  cost  on  the  Web,  and  whose  contents  can  be  obtained  for  a 
modest  fee  by  an  individual  or  library.  In  addition,  the  Collaboration  provides  materials  to  teach  individuals 
how  to  conduct  high-quality  systematic  reviews  and  meta-analyses. 

Cook  DJ,  Mulrow  CD,  Haynes  RB.  Systematic  reviews:  synthesis  of  best  evidence 
for  clinical  decisions.  Ann  Intern  Med,  1997;126:376-380. 

This  article  is  the  first  in  a  series  in  the  Annals  of  Internal  Medicine  on  systematic  reviews.  It  defines  a 
systematic  review  and  meta-analysis,  and  contrasts  the  systematic  approach  with  that  of  a  narrative  review, 
as  well  as  discussing  the  past  and  future  promise  of  systematic  reviews. 

Cook  DJ,  Sackett  D,  Spitzer  WO.  Methodologic  guidelines  for  systematic  reviews  of 
randomized  control  trials  in  health  care  from  the  Potsdam  Consultation  on  Meta¬ 
analysis.  J  Clin  Epidemiol,  1995;48:167-171. 

This  article  provides  guidelines  for  the  conducting  of  a  meta-analysis.  In  particular,  it  argues  that  similar  to 
a  randomized  controlled  trial,  a  meta-analysis  should  be  based  on  an  a  priori  protocol,  and  the  authors 
discuss  what  elements  should  be  defined  in  that  protocol. 

Cooper  H,  Hedges  LV,  eds.  The  Handbook  of  Research  Synthesis,  New  York:  Russell 
Sage  Foundation;  1994. 

This  book  provides  guidance  on  all  stages  of  a  literature  synthesis,  from  the  question  formulation,  to  study 
gathering,  to  the  pooling  and  presentation  of  results.  The  chapters  are  written  by  different  authors,  but 
common  examples  run  throughout  the  text.  The  book  is  not  written  from  the  perspective  of  medical 
applications,  but  contains  primarily  examples  from  education  and  psychology. 

Counsell  C.  Formulating  questions  and  locating  primary  studies  for  inclusion  in 
systematic  reviews.  Ann  Intern  Med,  1997;127(5):380-387, 

This  article  is  another  in  the  Annals  of  Internal  Medicine  series  on  systematic  reviews,  and  focuses  on  the 
formulation  of  the  question,  and  how  to  conduct  a  systematic,  unbiased,  and  comprehensive  search  of  the 
literature. 
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Egger  M,  Smith  GD,  Phillips  AN.  Meta-analysis:  principles  and  procedures.  BMJ. 
1997;315:1533-1537. 

This  article  is  one  of  a  series  in  BMJ  of  articles  on  methods  and  interpretation  of  meta-analysis.  It 
introduces  the  basic  steps  in  a  meta-analysis  for  the  nontechnical  reader. 

Egger  M,  Smith  GD,  Schneider  M,  Minder  C.  Bias  in  meta-analysis  detected  by  a 
simple,  graphical  test.  BMJ,  1997;315:629-634. 

This  article  is  another  in  the  BMJ  series.  It  discusses  the  use  of  funnel  plots,  in  which  each  study’s  effect 
size  is  plotted  versus  the  study’s  sample  size.  These  plots  may  be  useful  in  detecting  bias  in  meta-analyses. 

Egger  M,  Smith  GD.  Bias  in  location  and  selection  of  studies.  BMJ, 
1998;316:61-66. 

This  article  is  another  in  the  BMJ  series.  It  discusses  the  primary  data-gathering  step  in  a  systematic  review 
or  meta-analysis:  identifying  studies.  In  particular,  it  addresses  publication  bias,  and  the  bias  that  may  be 
introduced  by  the  exclusion  of  articles  not  published  in  English. 


Egger  M,  Zellweger-Zahner  T,  Schneider  M,  Junker  C,  Antes  G.  Language  bias  in 
randomized  controlled  trials  in  English  and  German.  Lancet,  1997;350:326-329. 

This  article  concludes  that  an  author  is  more  likely  to  publish  statistically  significant  results  from  a 
randomized  controlled  trial  in  an  English-language  journal,  and  thus  bias  may  result  if  a  systematic  review 
is  restricted  to  studies  published  in  English-language  journals. 

Hedges  LV,  Olkin  I.  Statistical  Methods  for  Meta-analysis,  New  York:  Academic 
Press,  Inc.;  1985. 

This  book  is  the  standard  reference  for  statistical  methods  in  meta-analysis.  The  text  is  technical.  It 
contains  several  data  examples  for  the  motivation  of  particular  techniques. 

Laird  NM,  Mosteller  F.  Some  statistical  methods  for  combining  experimental  results. 
Int  J  Technol  Assess  Health  Care,  1990;6:5-30. 

In  this  article,  the  authors  define  the  three  basic  meta-analytic  pooling  models:  equal  effects,  fixed  effects, 
and  random  effects.  They  apply  these  models  in  a  variety  of  settings  including  the  calculation  of  a  pooled 
proportion,  and  odds  ratio.  This  introduces  the  technical  issues  of  pooling  studies. 

Lau  J,  loannidis  PA,  Schmidt  CH.  Quantitative  synthesis  in  systematic  reviews. 

Ann  Intern  Med,  1997;127:820-826. 

This  article  is  another  in  the  Annals  of  Internal  Medicine  series  on  meta-analysis.  It  explains  basic  meta¬ 
analysis  methods  including  pooling  data  and  meta-regression,  and  is  a  good  introduction  for  the  nontechnical 
reader. 

Moher  D,  Jadad  AR,  Nichol  G,  Penman  M,  Tugwell  P,  Walsh  S.  Assessing  the 
quality  of  randomized  controlled  trials:  an  annotated  bibliography  of  scales  and 
checklists.  Control  Clin  Trials.  1995;16:62-73. 

This  article  presents  an  annotated  bibliography  of  twenty-five  scales  and  nine  checklists  that  have  been 
developed  to  assess  the  quality  of  randomized  controlled  trials. 
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Moher  D,  Pham  B,  Jones  A,  Cook  DJ,  Jadad  AR,  Moher  M,  Tugwell  P,  Klassen  TP. 
Does  the  quality  of  reports  of  randomized  trials  affect  estimates  of  intervention 
efficacy  reported  in  meta-analyses?  Lancet,  1998;352:609-613. 

The  authors  discuss  the  possible  relationship  between  study  quality  and  treatment  effect.  In  addition,  they 
provide  a  good  reference  list  to  other  publications  on  study  quality. 


Mulrow  C,  Langhorne  P,  Grimshaw  J.  Integrating  heterogeneous  pieces  of  evidence  in 
systematic  reviews.  Ann  Intern  Med,  1997;127:989-995. 

This  article  is  another  in  the  Annals  of  Internal  Medicine  series  on  systematic  reviews,  and  focuses  on  three 
topics:  (1)  how  reviewers  can  classify  heterogeneous  evidence;  (2)  what  factors  complicate  integration  of 
such  evidence;  and  (3)  what  strategies  can  be  used  to  integrate  such  evidence. 

Oxman  AD,  Cook  DJ,  Guyatt  GH.  Evidence-Based  Medicine  Working  Group.  Users’ 
guide  to  the  medical  literature.  VI.  How  to  use  an  overview.  JAMA, 
1994;272(17):1367-1371. 

This  article  is  one  in  the  JAMA  Users'  Guide  series  and  discusses  how  to  read  and  use  a  systematic  review 
or  meta-analysis.  As  compared  with  other  guideline  publications,  this  document  is  written  from  the 
perspective  of  a  clinician  caring  for  individual  patients,  rather  than  from  a  population-based  perspective. 

Sackett  DL,  Richardson  WS,  Rosenberg  W,  Haynes  RB.  Evidence^based  Medicine, 

New  York:  Churchill  Livingstone  Inc.;  1997. 

This  text  defines  evidence-based  medicine  and  explains  clearly  how  to  apply  its  key  principles.  The  book  is 
written  for  the  clinician.  It  gives  useful  examples  of  how  to  apply  evidence-based  medicine  in  one’s  own 
clinical  practice. 

U.S.  Preventive  Services  Task  Force.  Guide  to  Clinical  Preventive  Services,  2nd  ed. 
Baltimore,  Md.:  Williams  and  Wilkins;  1996. 

This  report  gives  examples  of  categorizing  evidence  on  the  efficacy  of  preventive  services  via  an  evidence 
hierarchy. 


Expert  Assessment 

Brook  RH,  The  RAND/UCLA  Appropriateness  Method,  Reprinted  from  McCormick 
KA,  Moore  SR,  and  Siegel  RA.  Methodology  Perspectives,  Rockville,  Md.:  Public 
Health  Service,  U.S.  Department  of  Health  and  Human  Services;  1994:59-70.  AHCR 
publication  95-0009.  (Also  available  as  RAND  reprint  RP-395.) 

This  document  describes  the  criteria  and  methods  used  in  the  development  of  the  RAND/UCLA 
Appropriateness  Method. 

Coulter  I,  Adams  A,  Shekelle  P.  Impact  of  varying  panel  membership  on  ratings  of 
appropriateness  in  consensus  panels:  a  comparison  of  multi-  and  single  disciplinary 
panel.  Health  Serv  Res,  1995;30:4:577-591. 

This  study  demonstrated  that  the  composition  of  a  panel  influences  the  ratings  of  various  medical 
indications.  In  addition,  it  was  found  that  those  who  use  a  given  procedure  are  more  likely  to  rate  it  as 
appropriate. 
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Ferguson  JH.  Technology  Transfer:  Consensus  and  Participation:  The  NIH  Consensus 
Development  Program,  Oakbrook  Terrace,  Ill.:  The  Joint  Commission  on 
Accreditation  of  Healthcare  Organizations;  July  1995.  Publication  1070-3241. 

This  document  describes  the  goals  of  the  NIH  Consensus  Development  Conferences  and  attempts  to  assess 
the  impact  of  the  conferences.  The  Office  of  Medical  Applications  Research  has  conducted  many  studies  on 
the  impact  of  the  consensus  conferences  and,  thus  far,  studies  have  shown  minimal  recognition  of  the  NIH 
program  or  conferences. 

Leape  LL,  Park  RE,  Kahan  JP,  Brook  RH.  Group  judgments  of  appropriateness:  the 
effect  of  panel  composition.  Qual  Assur  Health  Care,  1992  ;4(2):  15 1-159. 

In  this  study,  the  authors  compared  the  ratings  of  appropriateness  for  the  use  of  carotid  endarterectomy 
between  an  all-surgical  panel  and  a  multispecialty  panel.  It  was  found  that  the  surgical  panel  rated  more 
indications  appropriate  and  fewer  indications  inappropriate  than  the  multispecialty  panel. 

McGlynn  EA,  Kosecoff  J,  Brook  RH.  Format  and  conduct  of  consensus  development 
conferences:  multination  comparison.  Int  J  Technol  Assess  Health  Care, 
1990;6:450-469. 

This  document  compares  the  process  by  which  consensus  conferences  are  conducted  in  the  United  States, 
Canada,  Denmark,  Finland,  the  Netherlands,  Norway,  Sweden,  Switzerland,  and  the  United  Kingdom. 
Although  the  model  of  NIH  has  been  used  extensively  in  other  countries,  it  has  never  been  shown  to 
produce  replicable  results  or  to  be  preferable  to  other  models. 

Pearson  SD,  Margolis  CZ,  Davis  S,  Schreier  LK,  Sokol  HN,  Gottlieb  LK.  Is 
consensus  reproducible?  A  study  of  an  algorithmic  guidelines  development  process. 
Med  Care.  1995;33(6):643-660. 

The  objective  of  this  study  was  to  evaluate  the  reproducibility  of  a  consensus  guideline  development  process 
across  different  groups  of  physicians  in  a  health  maintenance  organization.  The  authors  demonstrate  that 
physician  panels  using  a  simplified  literature  support  and  consensus  process  may  produce  guidelines  that 
vary. 

Phelps  CE.  The  methodologic  foundations  of  studies  for  the  appropriateness  of 
medical  care.  New  Engl  J  Med,  1993;329(17):1241-1245. 

This  author  discusses  how  methods  used  to  study  appropriateness  can  lead  to  biased  estimates  of  the  rates  of 
inappropriate  treatment  and  that  these  rates  may  differ  markedly  from  the  true  rates.  The  overall  value  and 
credibility  of  methods  to  assess  the  appropriateness  of  medical  interventions  cannot  be  determined  until 
studies  estimate  the  sensitivity  and  specificity  of  this  “diagnostic  test.” 

Phelps  CE,  Mushlin  AI.  Focusing  technology  assessment  using  medical  decision 
theory.  Med  Beds  Making,  Oct-Dec  1988;8(4):279-289. 

The  authors  describe  a  hurdle  approach  to  assessments  of  diagnostic  technology.  The  first  hurdle  calculates 
the  cost-effectiveness  ratio  of  using  the  technology  assuming  that  it  is  perfectly  accurate.  If  the  technology 
is  worthy  under  that  assumption,  then  further  analysis  should  be  performed. 
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Shekelle  PG,  Kahan  JP,  Bernstein  SJ,  Leape  LL,  Kamberg  CJ,  Park  RE.  The 
reproducibility  of  a  method  to  identify  the  overuse  and  underuse  of  medical  procedures. 
New  Engl  J  Med,  1998;338:1888-1895. 

The  authors  conducted  a  scientifically  rigorous  test  of  the  reproducibility  of  the  RAND/UCLA 
Appropriateness  Method  and  found  that  when  the  method  is  used  to  measure  appropriateness  rates  in  a 
single  population,  precise  estimates  are  not  possible.  At  best,  in  a  single  population,  the  Appropriateness 
Method  can  estimate  whether  the  proportion  of  cases  with  overuse  is  small  or  large.  By  extension,  in 
making  decisions  for  individual  patients,  the  Appropriateness  Method  does  not  have  sufficient 
reproducibility  to  justify  its  use  as  a  gold  standard  of  appropriateness. 


Wortman  PM,  Smyth  JM,  Langenbrunner  JC,  Yeaton  WH.  Consensus  among  experts 
and  research  synthesis:  a  comparison  of  methods.  Int  J  Technol  Assess  Health  Care, 
1998;14(1):109-122. 

The  authors  find  a  positive  correlation  between  consensus  panels  and  systematic  review  and  synthesis 
within  the  context  of  assessing  carotid  endarterectomy.  The  authors  go  on  to  state:  “consensus  methods 
may  be  more  appropriate  during  the  early  or  ‘emerging’  phase  of  the  intervention  when  there  are  few  high- 
quality  research  studies  and  a  viable  research  synthesis  cannot  be  conducted.  As  the  intervention  develops 
and  sufficient  research  studies  accumulate,  research  synthesis  would  seem  more  appropriate.” 


Mathematical  Assessment 


Hogarth  RM,  Makridakis  S.  Forecasting  and  planning:  an  evaluation.  Manage 
ScL  1981;27(2):115-138. 

Reviews  the  psychological  literature  on  human  judgment  abilities  and  the  value  of  simple  quantitative 
models  in  overcoming  errors  in  judgment. 

Stokey  E,  Zeckhauser  R,  A  Primer  for  Policy  Analysis,  New  York:  Norton; 
1978. 

This  classic  book  is  intended  primarily  as  guide  for  public-sector  decisionmakers.  However,  this 
book  contains  chapters  on  modeling  approaches  that  are  relevant  for  health  care  decisionmakers,  such 
as  decision  analysis,  simulation,  Markov  models,  discounting,  and  linear  programming.  This  book  is 
technical,  although  the  authors  claim  that  readers  need  training  beyond  college  algebra.  This  book  has 
many  useful  examples. 


Cost-Outcome  Analysis 

Brown  M,  Fintor  L.  Cost-effectiveness  of  breast  cancer  screening:  preliminary 
results  of  a  systematic  review  of  the  literature.  Breast  Cancer  Res  Treat, 
1993;25(2):113-118. 

The  authors  review  16  studies  of  breast  cancer  screening  and  find  large  differences  in  the  reported  cost- 
effectiveness  of  screening.  The  authors  identify  the  different  assumptions  leading  to  the  disparate 
conclusions  made  in  two  of  these  studies. 
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Drummond  MF,  Richardson  WS,  O’Brien  JB,  Levine  M.  User’s  guide  to  the  medical 
literature:  XIII.  How  to  use  an  article  on  economic  analysis  of  clinical  practice^  A. 
are  the  results  of  the  study  valid?  JAMA.  May  21,  1997;277(19):1552-1557. 

This  is  the  first  of  two  articles  (see  O’Brien  et  al.,  1997)  written  for  clinicians  who  turn  to  cost-outcome 
analysis  to  assess  treatment  for  a  specific  patient.  Although  the  perspective  is  not  that  of  the  medical 
director,  the  discussion  is  an  excellent  introduction,  contains  many  references,  and  identifies  the  potential 
problems  with  cost-outcome  analyses. 

Eddy  D.  Oregon’s  methods:  did  cost-effectiveness  analysis  fail?  JAMA. 
1991;266:2135-2141. 

The  author  reviews  Oregon’s  attempted  practice  of  ranking  health  care  interventions  by  their  ratio  of  cost  to 
effect.  He  discusses  the  political  environment  and  its  implications  for  the  impact  of  cost-effectiveness 
studies. 

Gage  BF,  Carinalli  AB,  Albers  GW,  Owens  DK.  Cost-effectiveness  of  warfarin  and 
aspirin  for  prophylaxis  of  stroke  in  patients  with  nonvalvular  atrial  Hbrillation. 
JAMA.  1995;274(23):1839-1845. 

This  article  provides  an  example  of  cost-effectiveness  research  using  clinical  trial  data  and  decision  analysis. 
The  authors  examine  the  cost-effectiveness  of  treatment  with  warfarin,  aspirin,  or  no  treatment  for  the 
prevention  of  stroke.  The  authors  estimate  the  probability  of  stroke,  hemorrhage,  and  death  using 
published  clinical  trial  evidence,  and  perform  a  decision  analysis  to  evaluate  the  three  treatment  options. 


Gold  MR,  Siegel  JE,  Russell  LB,  Weinstein  MC.  Cost-Effectiveness  in  Health  and 
Medicine:  A  Report  of  the  US  Public  Health  Service,  New  York:  Oxford  University 
Press;  1996. 

This  book  reports  the  findings  of  an  expert  panel  convened  by  the  U.S.  Public  Health  service  to  study  cost- 
outcome  analyses.  It  contains  a  review  of  the  foundations,  the  current  practices,  and  the  current 
shortcomings/controversies  of  cost-outcome  analyses.  The  book  further  suggests  a  standardized  method  that 
had  been  previously  lacking.  The  findings  of  the  panel  are  widely  accepted  as  best  practices  for  cost- 
outcome  analyses.  One  assumption  of  the  book  that  will  make  strict  application  to  a  specific  health  plan 
difficult  is  that  the  panel  recommends  a  societal  perspective,  as  opposed  to  a  payer  perspective,  in 
performing  these  analyses. 

Haddix  AC,  Teusch  SM,  Shaffer  PA,  Dunet  DO.  Prevention  Effectiveness:  A  Guide 
to  Decision  Analysis  and  Economic  Evaluation  for  the  Prevention  Effectiveness 
Activity  Epidemiology  Program  Office,  Centers  for  Disease  Control  and  Prevention. 
New  York:  Oxford  University  Press;  1996. 

This  book  contains  an  easy-to-follow  description  of  how  to  perform  decision  analysis  and  cost-effectiveness 
analysis  as  suggested  by  a  Centers  for  Disease  Control  and  Prevention  course.  It  includes  several  useful 
examples  and  references. 

Laupacis  A,  Feeny  D,  Detsky  A,  Tugwell  P.  How  attractive  does  a  new  technology 
have  to  be  to  warrant  adoption  and  utilization?  Tentative  guidelines  for  using  clinical 
and  economical  evaluations.  Can  Med  Assn  J.  February  15,1992;  146(4):473-481. 

The  authors  suggest  a  grading  scheme  for  economic  evaluations  of  health  care  strategies.  The  authors  grade 
the  evidence  and  methods  used  in  studies,  as  well  as  the  ratio  of  cost  to  outcome.  They  recommend  grading 
based  on,  among  other  criteria,  the  ratio  of  cost  to  quality  adjusted  life  year. 


-63- 


Pharmaceutical  Technology  Assessment  for  Managed  Care 


Luce  B,  Elixhauser  A.  Estimating  costs  in  the  economic  evaluation  of  medical 
technologies.  Int  J  Technol  Assess  Health  Care,  1990;6:57-75. 

The  authors  explain  the  foundations  of  cost  estimation,  list  the  common  errors  made,  and  provide  a  list  of 
costs  to  consider. 

O’Brien  BJ,  Heyland  D,  Richardson  WS,  Levine  M,  Drummond  MF.  User’s  guide  to 
the  medical  literature:  XIII.  How  to  use  an  article  on  economic  analysis  of  clinical 
practice.  B.  What  are  the  results  and  will  they  help  me  in  caring  for  my  patients? 
JAMA.  June  11,  1997;277(22):1802-1806. 

This  is  the  second  of  two  articles  (see  Drummond,  1997)  written  for  clinicians  who  turn  to  cost-outcome 
analysis  to  assess  treatment  for  a  specific  patient.  As  with  Drummond,  the  perspective  is  not  that  of  the 
medical  director,  but  the  discussion  is  an  excellent  introduction,  contains  many  references,  and  identifies  the 
potential  problems  with  cost-outcome  analyses. 


Russell  LB,  Gold  MR,  Seigel  JE,  Daniels  N,  Weinstein  MC,  for  the  Panel  on  Cost- 
Effectiveness  in  Health  and  Medicine.  The  role  of  cost-effectiveness  analysis  in 
health  and  medicine.  JAMA,  1996;  276(14):1172-1177. 

This  article  is  the  first  in  a  three  part  series  that  summarizes  the  findings  of  the  Panel  on  Cost-Effectiveness 
in  Health  and  Medicine  convened  by  the  US  Public  Health  Service.  The  findings  of  the  panel  are  detailed  in 
Gold,  1996. 


Sloan  F,  ed.  Valuing  Health  Care:  Costs,  Benefits,  and  Effectiveness  of 
Pharmaceuticals  and  Other  Medical  Technologies,  New  York:  Cambridge  University 
Press;  1995. 

With  contributions  from  several  well-known  economic  analysts,  the  book  reviews  the  foundations  of  cost- 
outcome  analysis,  the  current  uses  of  cost-outcome  analysis,  and  recommends  best  practices. 


Tucker  AW,  Haddix  AC,  Bresee  JS,  Holman  RC,  Parashar  UD,  Glass  RI.  Cost- 
effectiveness  analysis  of  a  rotavirus  immunization  program  for  the  United  States. 
JAMA,  1998;279(17);1371-1376. 

An  example  of  cost-effectiveness  research  using  decision  analysis.  The  authors  examine  the  cost- 
effectiveness  of  a  US  rotavirus  immunization  program.  The  authors  used  a  decision  tree  to  estimate  the 
impact  of  the  proposed  program. 


Decision  Analysis 


Beck  J,  Pauker  S.  The  Markov  process  in  medical  prognosis.  Med  Decis  Making. 
1983;3:411-458. 

This  article  gives  a  nontechnical  introduction  to  Markov  processes  and  the  assumptions  of  these  models. 

Haddix  AC,  Teusch  SM,  Shaffer  PA,  Dunet  DO.  Prevention  Effectiveness:  A  Guide 
to  Decision  Analysis  and  Economic  Evaluation  for  the  Prevention  Effectiveness 
Activity  Epidemiology  Program  Office,  Centers  for  Disease  Control  and  Prevention. 
New  York:  Oxford  University  Press;  1996. 

This  book  contains  an  easy-to-follow  description  of  how  to  perform  decision  analysis  and  cost-effectiveness 
analysis  as  suggested  by  a  Centers  for  Disease  Control  and  Prevention  course.  It  includes  several  useful 
examples  and  references. 
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Law  A,  Kelton  W.  Simulation  Modeling  and  Analysis,  2nd  ed.  New  York:  McGraw- 
Hill;  1991. 

This  text  serves  as  a  reference  for  how  to  perform  and  analyze  the  results  of  simulation  analyses.  The  book 
is  written  for  persons  with  a  basic  familiarity  with  probability,  statistics,  and  computer  programming. 
Examples  in  the  book  are  drawn  from  a  wide  range  of  managerial  scenarios. 

Pauker  SG,  Kassirer  JP.  Decision  analysis.  New  Engl  J  Med.  1987:316(5):250-258. 

The  authors  describe  decision  analysis  to  physicians.  The  focus  is  on  clinical  decisionmaking  for  the 
individual.  The  discussion  is  quite  dated,  but  nicely  introduces  decision  analysis  and  Markov  processes. 

Richardson  SW,  Detsky  AS.  User’s  guide  to  the  medical  literature:  VIL  how  to  use  a 

clinical  decision  analysis,  A.  are  the  results  of  the  study  valid?  JAMA,  April  26, 

1995;273(16):1292-1295. 

This  is  the  first  of  two  articles  (see  Richardson,  May  1995)  written  for  clinicians  who  turn  to  decision 
analytic  studies  to  assess  treatment  for  a  specific  patient.  Although  the  perspective  is  not  of  the  medical 
director,  the  discussion  is  an  excellent  introduction,  contains  many  references,  and  identifies  the  potential 
problems  with  decision  analysis. 

Richardson  SW,  Detsky  AS.  User’s  guide  to  the  medical  literature:  VII.  how  to  use  a 

clinical  decision  analysis,  B.  what  are  the  results  and  will  they  help  me  in  caring  for 

my  patients?  JAMA.  May  24-31, 1995;273(20):1610-1613. 

This  is  the  second  of  two  articles  (see  Richardson,  April  1995)  written  for  clinicians  who  turn  to  decision 
analytic  studies  to  assess  treatment  for  a  specific  patient.  Although  the  perspective  is  not  of  the  medical 
director,  the  discussion  is  an  excellent  introduction,  contains  many  references,  and  identifies  the  potential 
problems  with  decision  analysis. 

Sox  HC,  Jr.,  Blatt  MA,  Higgins  MC,  Marton  KL.  Medical  Decision  Making. 

Boston,  Mass:  Butterworth-Heinemann;  1988. 

The  book  is  written  to  introduce  medical  practitioners  to  decision  analysis.  The  book  also  contains  an 
introduction  to  cost-outcome  analysis. 

Weinstein  M,  Fineberg  HV.  Clinical  Decision  Analysis,  Philadelphia,  Penn.:  WB 
Saunders  Company;  1980. 

A  classic  text  that  walks  the  reader  carefully  through  the  logic  and  perspective  of  decision  analysis. 


Demand  Modeling 


Duan  N,  Manning  WG,  Morris  CN,  Newhouse  JP.  A  comparison  of  alternative 
models  for  the  demand  for  medical  care.  J  Bus  Econ  Stat,  1983;1(2):115-126. 

A  classic  but  technical  article  about  the  use  of  econometric  models  to  estimate  demand  for  medical  care. 

The  framework  presented  in  the  model  is  useful  for  modeling  behavior,  such  as  initiation  of  pharmaceutical 
therapy  and  dosage  decisions. 
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Green  PE,  Krieger  AM.  An  application  of  a  product  positioning  model  to 
pharmaceutical  products.  Marketing  ScL  1992;11(2):117-132. 

Kontzalis  P.  Identification  of  key  attributes,  gap  analysis,  and  simulation  techniques 
in  forecasting  market  potential  of  ethical  pharmaceutical  products.  Int  J  Forecasting, 
1992;8:243-249. 

These  two  articles  lay  out  a  process  for  forecasting  demand  for  pharmaceuticals  through  a  process  of 
primary  data  collection,  modeling,  and  simulation.  The  articles  are  from  the  perspective  of  drug 
manufacturers.  However,  the  general  approach  is  useful  for  the  service  delivery  organizations  as  well.  The 
decisionmakers  considered  in  these  studies  are  hospital  dieticians  and  physicians,  respectively. 

Waldo  D.  Estimating  the  cost  of  the  Medicare  outpatient  drug  benefit.  Health  Care 
Financing  Rev.  1994;  15(3):  103-1 12. 

An  example  of  a  “back  of  the  envelope”  model  used  to  produce  cost  estimates  for  the  Clinton 
Administration’s  health  reform  initiative.  The  author  draws  parameters  from  a  range  of  sources  and 
performs  sensitivity  analysis  under  different  demand  scenarios. 


Primary  Data  Collection  for  Demand  Analysis 

McQuarrie  EF.  The  Market  Research  Tool  Box:  A  Concise  Guide  for  Beginners. 
Thousand  Oaks,  Calif.:  Sage  Publications;  1996. 

This  book  introduces  managers  to  a  range  of  marketing  research  methods.  The  author  discusses  the  value  of 
marketing  research  data  for  informing  business  decisions.  The  book  is  intended  as  a  background  text  rather 
than  a  “how-to”  guide.  The  author  includes  suggested  readings,  and  the  examples  are  intended  to  be  relevant 
to  managers  selling  expensive,  high-tech  products  but  also  inform  decisions  faced  by  medical  directors  and 
other  health  care  managers. 

Salant  P,  Dillman  DA.  How  to  Conduct  Your  Own  Survey.  New  York:  John  Wiley 
and  Sons;  1994. 

The  authors  present  an  applied  approach  to  developing  telephone  and  mail  surveys.  The  book  is  intended 
for  professionals  with  no  training  in  survey  research.  This  book  offers  systematic  instructions  on 
everything  from  question  and  questionnaire  design  to  budgeting.  A  strength  of  this  book  is  the  many 
practical  examples. 

Templeton  JF.  The  Focus  Group.  Chicago,  Ill.:  Probus  Publishing;  1994. 

A  practical  guide  to  organizing  and  conducting  focus  groups  and  analyzing  resulting  data.  This  book 
contains  sample  schedules,  interview  guides,  and  final  reports.  This  book  offers  advice  on  being  an  “ideal” 
client,  dealing  with  common  problems,  and  choosing  panelists. 
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Appendix  B:  Interview  Questionnaire 

DEFINITION  STATEMENTS; 


For  this  effort,  we  want  to  separate  consideration  of  “routine”  and  “revolutionary”  pharmacological  advances. 

•  Routine  is  defined  as  an  incremental  improvement  in  diagnosis  or  management  of  a  condition  that  is  currently 
handled  using  similar  approaches.  Examples  might  be  new  formulation  of  an  existing  drug  or  a  new  drug  with  a 
profile  similar  to  that  of  an  existing  drug. 

•  Revolutionary  involves  a  change  in  the  paradigm  for  diagnosis  or  management.  It  may  include  the  development 
of  medical  treatment  for  a  disease  formerly  managed  surgically  (and  vice  versa),  a  new  drug  which  is 
substantially  different  in  potency  or  mode  of  administration  or  action,  and  introduction  or  extension  of  drug 
therapy  to  persons  who  were  formerly  treated  expectantly  or  by  nonmedical  means. 


In  addition,  we  want  to  separate  consideration  of  “medical  necessity”  and  “coverage”  decisions. 

•  For  this  purpose,  medical  necessity  involves  decisions  regarding/identified  with  a  putative  clinical  need. 

•  Coverage  will  refer  to  decisions  intended  to  apply  to  larger  numbers  of  yet  unidentified  patients. 


THE  QUESTIONNAIRE 

A  WHATS  DONE  NOWIN  YOUR  ORGANIZATION 


•  In  regard  to  emerging  pharmaceuticals  or  new  indications  for  existing  drugs,  does  your  organization  make  rules 
and/or  decisions  internally ?( Y  /  N ) . .  ..probe  for  how  they  are  made. . . 

a.  National  vs.  local 

b.  Plan  vs.  PBM 

c.  Plan  vs.  Medical  Group 

d.  Coverage 

e.  Indications  for  use 

f .  Medical  necessity 

g.  Level  of  coverage  (co-payment)  for  drug 

h.  Formulary  inclusion 

•  For  those  areas  where  you  make  rules  and/or  decisions,  do  you  have  policies  and  procedures  in  place  and/or  a 
structured  process  for  considering  and  making  decisions? 


a. 

Coverage 

( Y/N) 

b. 

Indications  for  use 

(Y/N) 

c. 

Medical  necessity 

(Y/N) 

d 

Formulary  inclusion 

(Y/N) 
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•  Which  of  the  following  methods  are  employed  by  the  agent  who  performs  your  technology  assessment: 


(Elicit  distinctions  in  procedures  for  Medical  Necessity  vs.  Coverage 
decisions  and  for  Routine  vs.  Revolutionary  innovations.) 

a.  Expert  Judgment 

Formal  (consensus  or  guideline  panel,  e.g.,  Delphi) 

( Yes  No  Don  7  know/Unfamiliar  term) 

Informal  (individual  or  usual  committee,  e.g.,  P  &  T 
Or  expert  opinion) 

( Yes  No  Don  7  know/Unfamiliar  term) 


b.  Review  of  evidence 

•  Formal 

•  Literature  synthesis 

•  Meta-analysis 

•  Other 

•  Informal 

•  Casual  literature  review 

•  Other 


(Yes 

No 

Don't  know/Unfamiliar  term) 

(Yes 

No 

Don't  know/Unfamiliar  term) 

(Yes 

No 

Don't  know/Unfamiliar  term) 

(Yes 

No 

Don  7  know/Unfamiliar  term) 

(Yes 

No 

Don  7  know/Unfamiliar  term) 

c.  Modeling 
Formal 

•  Decision  analysis 

(Yes 

•  Cost-effectiveness/Cost-benefit  analysis 

(Yes 


No 

No 


Don  7  know/Unfamiliar  term) 
Don  7  know/Unfamiliar  term) 
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•  Marketing  models 

•  Other 

•  Informal 

•  Ad  hoc 

•  Back  of  the  envelope 

•  Other 


(Yes 

No 

Don't  know/Unfamiliar  term) 

(Yes 

No 

Don't  know/Unfamiliar  term) 

(Yes 

No 

Don't  know/Unfamiliar  term) 

(Yes 

No 

Don't  know/Unfamiliar  term) 

(Yes 

No 

Don't  know/Unfamiliar  term) 

•  Who  performs  the  technology  assessment  for  your  organization? _ 

If  the  assessment  is  performed  by  another  agent,  do  you  have  a  structured  process  to  participate  in  those  decisions? 

5.  How  does  your  organization  decide  on  when  to  review  an  emerging  drug  or  new  indications  for  use  or  to  what 
drugs  the  rules  should  be  applied?  Solicit  examples  for  illustration  purposes. 

a.  Coverage 

b.  How  and  when  it’s  covered  (prior  auth  /  indications  for  use  -  when  it’s  medically  necessaiy/special  benefit 
structure  /  formulary  preference) 

c.  If  drugs  fall  into  certain  categories,  are  they  treated  differently? 

—  Drugs  you  have  to  have  -  needed  to  treat  illness  and  injuiy 

—  Drugs  you  would  like  to  have  but  there  are  existing  /  already  approved  less  expensive  alternatives 
—  Drugs  that  are  lifestyle  focused 

—  Preventive  health  care  drugs  with  only  long-term  positive  impact 

6.  How  does  your  organization  learn  about  emerging  drugs,  classes  of  drugs,  or  new  indications  for  use? 

a.  What  mechanism  is  there  for  learning  about  emerging  drugs  in  the  pipeline 
—  Where  -  from  what  sources  -  do  you  learn  about  emerging  drugs? 

—  Is  there  a  formal  mechanism  for  surveillance. 

b.  What  mechanism  is  there  for  learning  about  new  indications  for  use? 

—  Wliere  -  from  what  sources  -  do  you  learn  about  new  indications  for  use? 

—  Is  there  a  formal  mechanism  for  surveillance? 
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7.  What  information  do  you  use  to  generate  your  decision  rules? 

Ask  if  they  are  willing  to  share  hard  copies  of  the  information,  process  flows,  etc. 

a.  What  sources  of  information  /  guidance  are  useful  in  decisionmaking  (including  specific  names  of  reference 
materials  they  recommend) 

b .  Role  of  HCFA  coverage  /  State  Medicaid  coverage 

c.  Role  of  competitors  approach  /  coverage 

d.  Role  of  consumer  request  or  direct-to-consumer  advertising 

8.  How  are  the  rules  applied  and  monitored  (e.g.,  indications  for  use)? 

a.  By  whom  are  the  rules  applied 

b.  How  is  the  effect  of  the  rules  measured 

R  WHATS  MISSING  TO  IMPROVE  THE  PROCESS 

1 .  What  would  you  like  to  have  that  you  don’t  have  to  support  the  processes  we  just  discussed? 

a.  Literature 

b .  Educational  Programs 

c.  Processes 

d  Models  -  financial  and  clinical  impact 

e.  Reference  materials 

Given  that  this  project  is  to  produce  a  reference  document  for  medical  directors  or  decisionmakers,  what  would  you 
suggest  would  be  useful  to  include  in  the  document? 
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